rbrooks/Quest-Board

Watch

Fork

You've already forked Quest-Board

Code Issues 58 Pull requests 7 Projects Releases 76 Packages Wiki Activity Actions

[Backend] Compose the summary from validated beats only, with a critic pass #334

New issue

Closed

opened 2026-08-25 20:38:53 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-08-25 20:38:53 +00:00

Collaborator

Copy link

Severity: HIGH. Found in the August 2026 session lifecycle review (#319).

Proposed fix

The final pass turns the validated, time-ordered beat list (~2-4k tokens) into the 3-6 paragraph prose summary, and may not introduce actors or events absent from the beats. Because the input is already ordered and grounded, this pass cannot reintroduce either failure mode.

Follow it with a cheap critic pass that diffs the prose against the beat list and strikes unsupported sentences. With beats in place this is nearly free, because every claim already has line references — as a bolt-on to the current architecture a critic would be far weaker, since claims without references force the checker to search the whole transcript.

The system prompt also needs fixing regardless of architecture: it currently contains no instruction to preserve chronology, and — unlike the highlights prompt — never explains what [HH:MM:SS] means or that the name before the colon is the speaker (audio_service.py:56-62). "Cover: key events and encounters, important NPC introductions..." actively invites thematic rather than chronological organisation.

Acceptance criteria

Compose input is the validated, sorted beat list, not the raw transcript
The prompt forbids introducing actors or events not present in the beats
A critic pass flags unsupported sentences
The summary optionally renders with timestamp anchors back into the transcript
The old single-shot path stays available behind a setting for one release

**Severity: HIGH.** Found in the August 2026 session lifecycle review (#319). ## Proposed fix The final pass turns the validated, time-ordered beat list (~2-4k tokens) into the 3-6 paragraph prose summary, and **may not introduce actors or events absent from the beats**. Because the input is already ordered and grounded, this pass cannot reintroduce either failure mode. Follow it with a cheap critic pass that diffs the prose against the beat list and strikes unsupported sentences. With beats in place this is nearly free, because every claim already has line references — as a bolt-on to the current architecture a critic would be far weaker, since claims without references force the checker to search the whole transcript. The system prompt also needs fixing regardless of architecture: it currently contains **no instruction to preserve chronology**, and — unlike the highlights prompt — never explains what `[HH:MM:SS]` means or that the name before the colon is the speaker (`audio_service.py:56-62`). "Cover: key events and encounters, important NPC introductions..." actively invites thematic rather than chronological organisation. ## Acceptance criteria - [ ] Compose input is the validated, sorted beat list, not the raw transcript - [ ] The prompt forbids introducing actors or events not present in the beats - [ ] A critic pass flags unsupported sentences - [ ] The summary optionally renders with timestamp anchors back into the transcript - [ ] The old single-shot path stays available behind a setting for one release

claude-bot added this to the v4.0.0 - Session Pipeline Accuracy milestone

2026-08-25 20:38:53 +00:00

claude-bot added the

bug

backend

labels

2026-08-25 20:38:53 +00:00

rbrooks referenced this issue from a commit

2026-08-26 01:30:27 +00:00

feat(backend): verify session summaries against the transcript (#332, #333, #334)

rbrooks referenced this issue from a commit

2026-08-28 01:38:24 +00:00

feat(backend): a critic pass over the composed summary (#334)

rbrooks referenced this issue from a commit

2026-08-28 18:39:30 +00:00

fix(backend): make the beat validator actually check what it claims (#332, #333)

rbrooks referenced this issue from a commit

2026-08-28 18:46:37 +00:00

fix(backend): stop the chunker recursing forever, and brief it on what it holds

rbrooks referenced this issue from a commit

2026-08-28 18:57:44 +00:00

feat(backend): anchors, a rollback lever, and a name the bot could not find

claude-bot referenced this issue

2026-08-28 19:14:22 +00:00

[Backend] Emit structured, timestamped beats with evidence references #332

claude-bot commented

2026-08-28 19:14:40 +00:00

Author

Collaborator

Copy link

Verified; the two unmet criteria built in 7a689a9, and a prompt bug fixed in ed441c8.

Criteria

Compose input is the validated, sorted beat list, not the raw transcript — true on the primary path, and now on the overflow sub-path too (below).
The prompt forbids introducing actors or events not present in the beats — COMPOSE_SYSTEM_PROMPT.
A critic pass flags unsupported sentences — and the count is now actually persisted.
The summary optionally renders with timestamp anchors back into the transcript — was not built. Now is.
The old single-shot path stays available behind a setting for one release — was not built. Now is.

`e992456`'s commit message was wrong about the critic count

It says the count "lands on SummarisationRecord, so it can be tracked like the other run metrics rather than only appearing in logs". It landed on the dataclass and went no further — there was no column and no write, so it existed only in logs, which is exactly what that sentence claims it avoids. Now on summarisation_runs.unsupported_sentences and in the API.

The beat list was being described to the model as a transcript

When a session produces more verified beats than compose can hold, the list is condensed first — and that pass was handed _WINDOW_SYSTEM_PROMPT, which tells the model it is reading '[HH:MM:SS] Speaker: text' dialogue and asks it not to speculate about what came before. A rendered beat list is [hh:mm:ss-hh:mm:ss] (kind) actors: summary.

So at the one moment the pipeline is already over budget, the model was briefed on the wrong document — which is how a condensed beat list turns into invented conversation. It now gets a prompt naming what it actually holds, and repeating that nothing may be dropped.

Timestamp anchors

(timestamp, what happened) pairs, chronological, one per verified beat, pointing at the beat's first cited line rather than t_start — after #333's re-anchoring those usually agree, and where they do not, the citation is the value that was checked.

Built in code, never asked of the model: this prompt explicitly forbids mentioning time ranges, and a model writing its own citations is what this milestone replaced. An anchor therefore cannot point at a line nobody spoke.

Unverified beats get none. An anchor is an invitation to go and look; sending a GM to a timestamp for an event the validator rejected spends their trust on the one claim already known to be unsupported. Those stay visible with their reasons on the runs API.

"Optionally" holds by construction — the summary text is untouched, and a client renders the anchors beside it or not.

The rollback lever

summarisation_mode: beats (default) or prose, following KEY_LORE_PIPELINE_MODE's pattern.

Deliberately not the same as the automatic fallback, which already covers "this provider cannot hold a JSON shape". This covers "the verified path is doing something wrong on my data and I need last release's behaviour while it is looked at" — a question a self-hoster cannot otherwise answer without editing code mid-campaign.

It defaults to the checked path; the GET endpoint states what choosing the other one gives up; and selecting it logs a warning both at the moment of choosing and on every session summarised under it, so nobody has to guess later why a session has no beats behind it. Expected to be removed once v4.0.0 has a release behind it, per the criterion's own "for one release".

Closing.

**Verified; the two unmet criteria built in `7a689a9`, and a prompt bug fixed in `ed441c8`.** ## Criteria - [x] **Compose input is the validated, sorted beat list, not the raw transcript** — true on the primary path, and **now on the overflow sub-path too** (below). - [x] **The prompt forbids introducing actors or events not present in the beats** — `COMPOSE_SYSTEM_PROMPT`. - [x] **A critic pass flags unsupported sentences** — and the count is now actually persisted. - [x] **The summary optionally renders with timestamp anchors back into the transcript** — **was not built.** Now is. - [x] **The old single-shot path stays available behind a setting for one release** — **was not built.** Now is. ## `e992456`'s commit message was wrong about the critic count It says the count *"lands on `SummarisationRecord`, so it can be tracked like the other run metrics rather than only appearing in logs"*. It landed on the dataclass and went no further — there was no column and no write, so it existed **only** in logs, which is exactly what that sentence claims it avoids. Now on `summarisation_runs.unsupported_sentences` and in the API. ## The beat list was being described to the model as a transcript When a session produces more verified beats than compose can hold, the list is condensed first — and that pass was handed `_WINDOW_SYSTEM_PROMPT`, which tells the model it is reading `'[HH:MM:SS] Speaker: text'` dialogue and asks it not to speculate about what came before. A rendered beat list is `[hh:mm:ss-hh:mm:ss] (kind) actors: summary`. So at the one moment the pipeline is already over budget, the model was briefed on the wrong document — which is how a condensed beat list turns into invented conversation. It now gets a prompt naming what it actually holds, and repeating that nothing may be dropped. ## Timestamp anchors `(timestamp, what happened)` pairs, chronological, one per **verified** beat, pointing at the beat's *first cited line* rather than `t_start` — after #333's re-anchoring those usually agree, and where they do not, the citation is the value that was checked. Built in code, never asked of the model: this prompt explicitly forbids mentioning time ranges, and a model writing its own citations is what this milestone replaced. An anchor therefore cannot point at a line nobody spoke. Unverified beats get none. An anchor is an invitation to go and look; sending a GM to a timestamp for an event the validator rejected spends their trust on the one claim already known to be unsupported. Those stay visible with their reasons on the runs API. "Optionally" holds by construction — the summary text is untouched, and a client renders the anchors beside it or not. ## The rollback lever `summarisation_mode`: `beats` (default) or `prose`, following `KEY_LORE_PIPELINE_MODE`'s pattern. Deliberately **not** the same as the automatic fallback, which already covers "this provider cannot hold a JSON shape". This covers "the verified path is doing something wrong on my data and I need last release's behaviour while it is looked at" — a question a self-hoster cannot otherwise answer without editing code mid-campaign. It defaults to the checked path; the GET endpoint states what choosing the other one gives up; and selecting it logs a warning both at the moment of choosing **and on every session summarised under it**, so nobody has to guess later why a session has no beats behind it. Expected to be removed once v4.0.0 has a release behind it, per the criterion's own "for one release". Closing.

claude-bot closed this issue

2026-08-28 19:16:06 +00:00

claude-bot referenced this issue

2026-09-09 04:47:35 +00:00

The unsupported-sentence critic counts sentence-opening words as unknown names #567

rbrooks referenced this issue from a commit

2026-09-10 05:32:39 +00:00

feat(webapp): a bounded summary and a complete event log (#568)

rbrooks referenced this issue from a commit

2026-09-10 05:49:55 +00:00

feat(webapp): a bounded summary and a complete event log (#568)

No Branch/Tag specified

main

feat/1328-fill-it-in

feat/1328-fill-frontend

feat/1328-fill-backend

feat/9-foundry-push

fix/9-foundry-live

feat/9-foundry-push-frontend

renovate/celery-5.x

feat/5-foundry-connect

feat/9-foundry-push-backend

feat/9-foundry-push-module

feat/5-foundry-frontend

feat/5-foundry-backend

feat/5-foundry-module

release/v4.8.0-rc1

docs/d341-convert-draft

feat/958-update-to-version

feat/958-convert-sheet

feat/958-pathbuilder-import-frontend

fix/1360-heading-order

feat/958-pathbuilder-import-backend

feat/11-weekly-digest

feat/1341-auto-update

docs/d332-superseded

chore/1345-names-cleanup

feat/1342-mentioned-pages

feat/1328-recipes

fix/1346-pgdg-key

perf/1247-shared-recording-stream

chore/132-gitleaks-allowlist

release/v4.7.1

spike/foundry-module

release/v4.7.1-rc1

docs/decisions-d332-d337

fix/bot-certifi

chore/release-4.7.0

chore/release-4.7.0-rc4

chore/deps-4.7.0

chore/release-4.7.0-rc3

fix/1280-quote-must-say-it

fix/1257-names-row-squeeze

chore/release-4.7.0-rc2

fix/1245-pg-upgrade-count

chore/release-4.7.0-rc1

feat/1257-names-in-prep

feat/1257-be

feat/1257-fe

feat/1257-rating-fixes

design/1328-fill-it-in

feat/1257-name-splits

fix/1326-redis-503

feat/1280-fact-suggestions

feat/1280-be

feat/1280-fe

design/1280-fact-suggestions

feat/1279-infobox

feat/1279-be

feat/1279-fe

feat/1321-shelf-copies

design/1279-infobox

feat/1243-table-find

feat/1257-who-rated

feat/1243-prep-drawer

feat/1243-sheet-pickers

design/1243-browse-in-place

feat/1315-hotwords-default

feat/1242-system-filters

feat/1257-more-name-styles

feat/506-leg3-hotwords

design/1242-system-filters

fix/1302-packs-polish

feat/1241-rules-library

design/1241-rules-library

feat/1245-postgres-18

integ/4.6.2

fix/1263-map-ui-2

fix/1263-map-server

fix/1263-map-ui-1

fix/1261-editor-link-menu

fix/1229-axe-4.14-name-mismatch

renovate/axllent-mailpit-1.x

renovate/eslint-monorepo

renovate/authlib-1.x

chore/release-4.6.0

chore/release-4.6.0-rc2

fix/460-rc2-screens

fix/460-rc2-server

fix/release-prerelease-exit

chore/release-4.6.0-rc1

feat/125-campaign-templates

feat/126-calendar

feat/1181-pictures-and-files

feat/1190-event-people

integ/4.6.0

feat/1191-line-of-the-night

feat/1200-starter-packs

chore/bot-multidict-and-playwright-group

fix/1202-this-week-answered-count

fix/122-124-leftovers

fix/1178-get-routes-that-write

fix/1203-journeys-check-names

design/125-campaign-templates

feat/126-calendar-engine

design/126-in-world-calendar

fix/1189-cited-stamp-in-range

fix/1189-v4-looser-check

fix/1189-richer-beats

fix/1113-export-bind-lists

fix/1154-release-notes-by-file

chore/release-4.5.0

chore/release-4.5.0-rc12

fix/1135-wiki-markup-as-text

fix/demo-usage-cost-in-window

chore/release-4.5.0-rc11

fix/1121-sheet-consistency

fix/1121-sheet-5e-terms-and-card-hp

copy/1122-busiest-pages

fix/1121-relevant-choices

fix/1121-wiki-create-npc-save-sheet-links

fix/1029-menu-stays-on-screen

docs/prototype-transcript-bin-wording

fix/1023-editor-save-row-back-scroll

fix/1020-search-exact-first

fix/1020-table-prep

fix/1020-shell-wordmark-footer-bell

fix/1020-wiki-article-party

fix/872-admin-kept-until-phone-tables

feat/l13-milestone-level-waiting

fix/1000-1001-bell-older-setup-context

fix/999-pronouns-in-prompts

fix/l19-other-class-feats

fix/1010-transcript-window-ordinary-confirm

docs/d254-d269-decisions

fix/702-member-search

fix/680-transcript-bin

fix/993-unidentified-prompts

fix/841-redis-timeouts

fix/990-binned-not-exported

fix/830-export-prep-shelf

fix/deps-audit-2026-10-01

fix/877-title-reads

fix/798-audio-received-at

build/lane11-17-j7

build/lane11-16-levelup

build/lane11-14-seams

build/lane11-14-seed

build/lane11-15-start

build/lane11-13-old-block-out

fix/980-review-r3

build/lane11-12-block

build/lane11-8-packs

fix/demo-seed-canonical-names

build/lane11-11-hot-half

docs/l11-decisions

build/lane11-7-convert

build/lane11-10-editing

fix/npm-audit-brace-expansion

build/lane11-9-sheet-page

fix/bot-pyjwt

build/lane11-6-content

fix/lane11-rank-names

build/lane11-2b-language

build/lane11-4-js-evaluator

build/lane11-3-server

build/lane11-5a-pf2e

build/lane11-5b-5e

feat/design-admin

build/lane11-2-evaluator

build/lane11-1-sheet-pieces

build/lane11-0-drawing

fix/943-quest-objectives-fk

fix/prepsheet-shelf-flake

fix/945-leftovers

build/demo-gm-copy

fix/901-email-landings

build/lane10-8-journeys

build/demo-seed-lanes-8-10

fix/887-gm-own-confirmation

build/lane9-5-j1-admin

fix/918-audience-click

fix/useasync-overtaken

ci/939-critical-path

test/937-backend-suite-speed

chore/claude-orchestration

fix/924-xp-gm-only

fix/session-stale-poll

docs/decisions-d239-d242

fix/930-pypdf

build/lane8-journeys

build/925-quest-editor

fix/894-private-page-pictures

build/lane8-p4-drafts

build/lane8-p7-map

build/lane8-p6-handouts-rules

fix/wiki-fixture-names

build/lane10-d227-test-usage

fix/rsvp-horizon-test-tuesday

fix/media-volume-owner

build/lane8-p3-editor

fix/912-unidentified-rename

build/lane8-p5-party-quests

build/lane8-handout-takeback

build/lane10-pickers

build/lane8-p2-article

build/lane8-6-map

build/lane10-7b-email-pages

build/lane8-5-handouts

build/lane10-7a-test-the-bot-pages

build/lane8-p1-pages

build/lane8-4-quests

build/lane8-2b-feeds

build/lane8-3-parties

build/lane8-2a-links

build/lane10-5-commands

build/lane9-6-seed

build/lane10-4b-reactions

build/lane9-4d-pages

build/lane9-4c-pages

build/lane10-4a-posts

build/lane10-3-email

fix/891-web-fonts

build/lane9-4b-pages

build/lane10-6-test-the-bot

build/lane10-2-notify-out

build/lane8-1-wiki-text

build/lane9-4a-frame

design/lane10-drawings

fix/j6-budget

fix/j5-phone-flake

fix/869-test-leaks

fix/843-record-start-followups

build/lane7-all

build/lane6-all

build/lane9-b2-server

chore/changelog-fragments

docs/decisions-d218

build/lane9-a-owner-overview

fix/rule-card-level-label

fix/834-retention-semantics

fix/o3c-record-without-announcement

docs/decisions-d179

docs/decisions-d131-d170

fix/851-commit-before-response

build/lane6-j3-seed

build/lane9-b1-d67-identity

fix/841-blocking-loop

build/lane9-c-backups-audit

fix/vote-race-and-start-commit

fix/recording-announcement

fix/proposal-read-leak

fix/last-admin-guard

build/lane6-p4c-planning

build/lane6-p4b-margin

build/lane6-p4a-sheet

build/lane5-j4-seed

build/lane6-base

build/lane7-pages-talk

build/lane5-pages

build/lane7-pages-safety

build/lane7-p3-safety

build/lane7-p2-notify

build/lane6-p3-planning

build/lane6-p2-history

build/lane6-p1b-items

build/lane7-p1-threads

build/lane5-p6-faces

build/lane6-p1a-sheet

build/lane5-p5-find

build/lane5-p4a-board

build/lane5-p4b-scenes

build/lane5-p3-live

build/lane5-base

build/lane5-p2-cards

docs/lane5-decisions

build/review-r2-wide

fix/review-r1

build/lane5-p1-scenes

build/demo-seed

build/lane4-journeys-j5

build/lane4-page-review-b

build/lane4-page-review-a

build/lane4-page-read

build/lane4-journeys

build/lane4-page-sess2

build/lane4-page-sess1

build/lane4-notes

build/lane4-discord-on-approval

build/lane4-scheduling

build/lane4-versions

build/760-session-number

build/lane4-decisions

build/797-erase-no-regenerate

build/test-race-audit

build/lane3-completions

build/794-dark-only

build/796-titles-scroll

build/791-retention-anchor

build/a11y-sweep

build/shell-followups

build/retention-preview

build/753-backup-readback

docs/d91-d93

build/lane3-j1

build/ci-harness-paths-selftest

fix/useauth-test-race

build/ci-harness-paths

build/777-ai-settings-read

build/lane3-settings-c

build/lane3-settings-b

build/lane3-j6

build/lane3-sessionform

build/lane3-settings-a

build/lane3-members

build/lane3-thisweek

build/lane3-story

chore/87-align-python-3.12

v4.8.0-rc1

v4.7.1

v4.7.1-rc1

v4.7.0

v4.7.0-rc4

v4.7.0-rc3

v4.7.0-rc2

v4.7.0-rc1

v4.6.2

v4.6.2-rc4

v4.6.2-rc3

v4.6.2-rc2

v4.6.2-rc1

v4.6.1

v4.6.1-rc1

v4.6.0

v4.6.0-rc2

v4.6.0-rc1

v4.5.3

v4.5.3-rc1

v4.5.2

v4.5.2-rc1

v4.5.1

v4.5.1-rc2

v4.5.1-rc1

v4.5.0

v4.5.0-rc12

v4.5.0-rc11

v4.5.0-rc10

v4.5.0-rc9

v4.5.0-rc8

v4.5.0-rc7

v4.5.0-rc6

v4.5.0-rc5

v4.5.0-rc4

v4.5.0-rc3

v4.5.0-rc2

v4.5.0-rc1

v4.4.1

v4.4.0

v4.3.0

v4.2.3

v4.2.2

v4.2.1

v4.2.0

v4.1.2

v4.1.1

v4.1.0

v4.0.1

v4.0.0

v3.11.4

v3.11.3

v3.11.2

v3.11.1

v3.11.0

v3.10.0

v3.9.1

v3.9.0

v3.8.0

v3.7.0

v3.6.1

v3.6.0

v3.5.0

v3.4.1

v3.4.0

v3.3.0

v3.2.0

v3.1.0

v3.0.0

v2.1.0

v2.0.0

v1.1.0

v1.0.0

v0.12.0

v0.11.0

v0.10.0

v0.9.1

v0.3.3

v0.9.0

v0.8.0

v0.3.2

v0.3.1

v0.3.0

v0.7.0

v0.6.0

v0.5.0

v0.4.0

v0.2.1

v0.2.0

v0.1.1

v0.1.0

Labels

Clear labels

backend

Issue related to the backend

bot

Issue related to the bot

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

frontend

Issue related to the frontend

More information is needed

recording

Issue related to the recording system

security

Security hardening / vulnerability work

No labels

Milestone

Clear milestone

No items

No milestone

v4.0.0 - Session Pipeline Accuracy

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/Quest-Board#334

Reference in a new issue

Repository

rbrooks/Quest-Board

Title

Body

No description provided.

Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?

Rows
Columns

[Backend] Compose the summary from validated beats only, with a critic pass #334

Proposed fix

Acceptance criteria

Criteria

e992456's commit message was wrong about the critic count

The beat list was being described to the model as a transcript

Timestamp anchors

The rollback lever

`e992456`'s commit message was wrong about the critic count