[Backend] Compose the summary from validated beats only, with a critic pass

claude-bot commented

2026-08-25 20:38:53 +00:00

Contributor

Severity: HIGH. Found in the August 2026 session lifecycle review (#319).

Proposed fix

The final pass turns the validated, time-ordered beat list (~2-4k tokens) into the 3-6 paragraph prose summary, and may not introduce actors or events absent from the beats. Because the input is already ordered and grounded, this pass cannot reintroduce either failure mode.

Follow it with a cheap critic pass that diffs the prose against the beat list and strikes unsupported sentences. With beats in place this is nearly free, because every claim already has line references — as a bolt-on to the current architecture a critic would be far weaker, since claims without references force the checker to search the whole transcript.

The system prompt also needs fixing regardless of architecture: it currently contains no instruction to preserve chronology, and — unlike the highlights prompt — never explains what [HH:MM:SS] means or that the name before the colon is the speaker (audio_service.py:56-62). "Cover: key events and encounters, important NPC introductions..." actively invites thematic rather than chronological organisation.

Acceptance criteria

Compose input is the validated, sorted beat list, not the raw transcript
The prompt forbids introducing actors or events not present in the beats
A critic pass flags unsupported sentences
The summary optionally renders with timestamp anchors back into the transcript
The old single-shot path stays available behind a setting for one release

**Severity: HIGH.** Found in the August 2026 session lifecycle review (#319). ## Proposed fix The final pass turns the validated, time-ordered beat list (~2-4k tokens) into the 3-6 paragraph prose summary, and **may not introduce actors or events absent from the beats**. Because the input is already ordered and grounded, this pass cannot reintroduce either failure mode. Follow it with a cheap critic pass that diffs the prose against the beat list and strikes unsupported sentences. With beats in place this is nearly free, because every claim already has line references — as a bolt-on to the current architecture a critic would be far weaker, since claims without references force the checker to search the whole transcript. The system prompt also needs fixing regardless of architecture: it currently contains **no instruction to preserve chronology**, and — unlike the highlights prompt — never explains what `[HH:MM:SS]` means or that the name before the colon is the speaker (`audio_service.py:56-62`). "Cover: key events and encounters, important NPC introductions..." actively invites thematic rather than chronological organisation. ## Acceptance criteria - [ ] Compose input is the validated, sorted beat list, not the raw transcript - [ ] The prompt forbids introducing actors or events not present in the beats - [ ] A critic pass flags unsupported sentences - [ ] The summary optionally renders with timestamp anchors back into the transcript - [ ] The old single-shot path stays available behind a setting for one release

claude-bot added this to the v4.0.0 - Session Pipeline Accuracy milestone

2026-08-25 20:38:53 +00:00

claude-bot added the

bug

backend

labels

2026-08-25 20:38:53 +00:00

rbrooks referenced this issue from a commit

2026-08-26 01:30:27 +00:00

feat(backend): verify session summaries against the transcript (#332, #333, #334)

rbrooks referenced this issue from a commit

2026-08-28 01:38:24 +00:00

feat(backend): a critic pass over the composed summary (#334)

rbrooks referenced this issue from a commit

2026-08-28 18:39:30 +00:00

fix(backend): make the beat validator actually check what it claims (#332, #333)

rbrooks referenced this issue from a commit

2026-08-28 18:46:37 +00:00

fix(backend): stop the chunker recursing forever, and brief it on what it holds

rbrooks referenced this issue from a commit

2026-08-28 18:57:44 +00:00

feat(backend): anchors, a rollback lever, and a name the bot could not find

claude-bot referenced this issue

2026-08-28 19:14:22 +00:00

[Backend] Emit structured, timestamped beats with evidence references #332

claude-bot commented

2026-08-28 19:14:40 +00:00

Author

Contributor

Verified; the two unmet criteria built in 7a689a9, and a prompt bug fixed in ed441c8.

Criteria

Compose input is the validated, sorted beat list, not the raw transcript — true on the primary path, and now on the overflow sub-path too (below).
The prompt forbids introducing actors or events not present in the beats — COMPOSE_SYSTEM_PROMPT.
A critic pass flags unsupported sentences — and the count is now actually persisted.
The summary optionally renders with timestamp anchors back into the transcript — was not built. Now is.
The old single-shot path stays available behind a setting for one release — was not built. Now is.

`e992456`'s commit message was wrong about the critic count

It says the count "lands on SummarisationRecord, so it can be tracked like the other run metrics rather than only appearing in logs". It landed on the dataclass and went no further — there was no column and no write, so it existed only in logs, which is exactly what that sentence claims it avoids. Now on summarisation_runs.unsupported_sentences and in the API.

The beat list was being described to the model as a transcript

When a session produces more verified beats than compose can hold, the list is condensed first — and that pass was handed _WINDOW_SYSTEM_PROMPT, which tells the model it is reading '[HH:MM:SS] Speaker: text' dialogue and asks it not to speculate about what came before. A rendered beat list is [hh:mm:ss-hh:mm:ss] (kind) actors: summary.

So at the one moment the pipeline is already over budget, the model was briefed on the wrong document — which is how a condensed beat list turns into invented conversation. It now gets a prompt naming what it actually holds, and repeating that nothing may be dropped.

Timestamp anchors

(timestamp, what happened) pairs, chronological, one per verified beat, pointing at the beat's first cited line rather than t_start — after #333's re-anchoring those usually agree, and where they do not, the citation is the value that was checked.

Built in code, never asked of the model: this prompt explicitly forbids mentioning time ranges, and a model writing its own citations is what this milestone replaced. An anchor therefore cannot point at a line nobody spoke.

Unverified beats get none. An anchor is an invitation to go and look; sending a GM to a timestamp for an event the validator rejected spends their trust on the one claim already known to be unsupported. Those stay visible with their reasons on the runs API.

"Optionally" holds by construction — the summary text is untouched, and a client renders the anchors beside it or not.

The rollback lever

summarisation_mode: beats (default) or prose, following KEY_LORE_PIPELINE_MODE's pattern.

Deliberately not the same as the automatic fallback, which already covers "this provider cannot hold a JSON shape". This covers "the verified path is doing something wrong on my data and I need last release's behaviour while it is looked at" — a question a self-hoster cannot otherwise answer without editing code mid-campaign.

It defaults to the checked path; the GET endpoint states what choosing the other one gives up; and selecting it logs a warning both at the moment of choosing and on every session summarised under it, so nobody has to guess later why a session has no beats behind it. Expected to be removed once v4.0.0 has a release behind it, per the criterion's own "for one release".

Closing.

**Verified; the two unmet criteria built in `7a689a9`, and a prompt bug fixed in `ed441c8`.** ## Criteria - [x] **Compose input is the validated, sorted beat list, not the raw transcript** — true on the primary path, and **now on the overflow sub-path too** (below). - [x] **The prompt forbids introducing actors or events not present in the beats** — `COMPOSE_SYSTEM_PROMPT`. - [x] **A critic pass flags unsupported sentences** — and the count is now actually persisted. - [x] **The summary optionally renders with timestamp anchors back into the transcript** — **was not built.** Now is. - [x] **The old single-shot path stays available behind a setting for one release** — **was not built.** Now is. ## `e992456`'s commit message was wrong about the critic count It says the count *"lands on `SummarisationRecord`, so it can be tracked like the other run metrics rather than only appearing in logs"*. It landed on the dataclass and went no further — there was no column and no write, so it existed **only** in logs, which is exactly what that sentence claims it avoids. Now on `summarisation_runs.unsupported_sentences` and in the API. ## The beat list was being described to the model as a transcript When a session produces more verified beats than compose can hold, the list is condensed first — and that pass was handed `_WINDOW_SYSTEM_PROMPT`, which tells the model it is reading `'[HH:MM:SS] Speaker: text'` dialogue and asks it not to speculate about what came before. A rendered beat list is `[hh:mm:ss-hh:mm:ss] (kind) actors: summary`. So at the one moment the pipeline is already over budget, the model was briefed on the wrong document — which is how a condensed beat list turns into invented conversation. It now gets a prompt naming what it actually holds, and repeating that nothing may be dropped. ## Timestamp anchors `(timestamp, what happened)` pairs, chronological, one per **verified** beat, pointing at the beat's *first cited line* rather than `t_start` — after #333's re-anchoring those usually agree, and where they do not, the citation is the value that was checked. Built in code, never asked of the model: this prompt explicitly forbids mentioning time ranges, and a model writing its own citations is what this milestone replaced. An anchor therefore cannot point at a line nobody spoke. Unverified beats get none. An anchor is an invitation to go and look; sending a GM to a timestamp for an event the validator rejected spends their trust on the one claim already known to be unsupported. Those stay visible with their reasons on the runs API. "Optionally" holds by construction — the summary text is untouched, and a client renders the anchors beside it or not. ## The rollback lever `summarisation_mode`: `beats` (default) or `prose`, following `KEY_LORE_PIPELINE_MODE`'s pattern. Deliberately **not** the same as the automatic fallback, which already covers "this provider cannot hold a JSON shape". This covers "the verified path is doing something wrong on my data and I need last release's behaviour while it is looked at" — a question a self-hoster cannot otherwise answer without editing code mid-campaign. It defaults to the checked path; the GET endpoint states what choosing the other one gives up; and selecting it logs a warning both at the moment of choosing **and on every session summarised under it**, so nobody has to guess later why a session has no beats behind it. Expected to be removed once v4.0.0 has a release behind it, per the criterion's own "for one release". Closing.

claude-bot closed this issue

2026-08-28 19:16:06 +00:00

claude-bot referenced this issue

2026-09-09 04:47:35 +00:00

The unsupported-sentence critic counts sentence-opening words as unknown names #567

rbrooks referenced this issue from a commit

2026-09-10 05:32:39 +00:00

feat(webapp): a bounded summary and a complete event log (#568)

rbrooks referenced this issue from a commit

2026-09-10 05:49:55 +00:00

feat(webapp): a bounded summary and a complete event log (#568)