Session events are too terse: the extraction prompt asks for one bare sentence

claude-bot commented

2026-10-14 04:18:39 +00:00

Collaborator

Owner, 13 Oct 2026, reviewing Session 14 on prod: "the events in particular have a lot less detail and are extremely brief."

Cause. beat_service.EXTRACT_SYSTEM_PROMPT describes each beat's summary only as "<one sentence, plain past tense>". Every other rule is about accuracy (citations, actors, what is not story). The current model (Gemma 4, served under the alias qwen3.5) honours "one sentence" minimally, and the summary is composed from these beats, so it thins too. Session 14's run: 86 beats extracted, 76 validated, 4 windows, summary 2,213 characters; nothing failed or truncated.

Change. Ask for specifics in each beat: one or two sentences, who did what, to whom or with what, and the result, using the names of characters, places and items as said in the transcript; an example of specific versus general. Keep every accuracy rule.

Risk. More detail invites invented detail. Validation checks a beat's citations and actors, not every noun in its sentence.

Evidence before shipping. Run the eval harness (webapp/backend/evals) with the live provider on the real configured model, current prompt against new, on the committed fixtures including the synthetic session: coverage, attribution, chronology, beat validation and precision must not fall, and a detail measure (e.g. mean words per beat, named entities per beat that appear in the cited lines vs. that do not) should rise. Record both runs (--record).

Then, on the owner's choice, regenerate Session 14's summary on prod to compare on a real session.

Owner, 13 Oct 2026, reviewing Session 14 on prod: "the events in particular have a lot less detail and are extremely brief." **Cause.** `beat_service.EXTRACT_SYSTEM_PROMPT` describes each beat's `summary` only as `"<one sentence, plain past tense>"`. Every other rule is about accuracy (citations, actors, what is not story). The current model (Gemma 4, served under the alias `qwen3.5`) honours "one sentence" minimally, and the summary is composed from these beats, so it thins too. Session 14's run: 86 beats extracted, 76 validated, 4 windows, summary 2,213 characters; nothing failed or truncated. **Change.** Ask for specifics in each beat: one or two sentences, who did what, to whom or with what, and the result, using the names of characters, places and items as said in the transcript; an example of specific versus general. Keep every accuracy rule. **Risk.** More detail invites invented detail. Validation checks a beat's citations and actors, not every noun in its sentence. **Evidence before shipping.** Run the eval harness (`webapp/backend/evals`) with the live provider on the real configured model, current prompt against new, on the committed fixtures including the synthetic session: coverage, attribution, chronology, beat validation and precision must not fall, and a detail measure (e.g. mean words per beat, named entities per beat that appear in the cited lines vs. that do not) should rise. Record both runs (`--record`). Then, on the owner's choice, regenerate Session 14's summary on prod to compare on a real session.

claude-bot referenced this issue

2026-10-14 04:20:36 +00:00

Review events: the GM can change who was involved when rewording an event #1190

rbrooks referenced this issue from a commit

2026-10-14 05:07:02 +00:00

evals: report how specific the beats are; richer-beat prompt not shipped (#1189)

claude-bot commented

2026-10-14 05:08:05 +00:00

Author

Collaborator

First round (branch fix/1189-richer-beats, commit ee928508; numbers in evals/results/history.jsonl). Live against Gemma 4 26B (alias qwen3.5), 6 synthetic fixtures + Halloway Reach, current prompt ×4 and four variants ×2-3.

prompt	coverage	attribution	beat validation	words/beat	invented names
current	0.751	0.515	0.840	10.9	0
v4 (best)	0.714	0.853	0.733	15.8	0

Every variant made beats longer and specific with no invented names, but beat validation and coverage fell beyond the current prompt's spread, so no prompt change shipped. Causes from the failed beats: a two-actor example was copied; "to whom" moved the person acted on into actors (fixed in v3, which is why attribution rose); "and how it turned out" made the model fold the GM's set-up and outcome into the player's action and cite the GM's lines instead of the player's, which fails validation and merges scene beats away (74 → 52 verified).

Shipped on the branch: an eval detail report (evals/detail.py, --detail): words per beat and names grounded in cited lines / elsewhere / invented.

Next: one more round, v5 = v4 without the outcome clause.

First round (branch `fix/1189-richer-beats`, commit ee928508; numbers in `evals/results/history.jsonl`). Live against Gemma 4 26B (alias `qwen3.5`), 6 synthetic fixtures + Halloway Reach, current prompt ×4 and four variants ×2-3. | prompt | coverage | attribution | beat validation | words/beat | invented names | |---|---|---|---|---|---| | current | 0.751 | 0.515 | 0.840 | 10.9 | 0 | | v4 (best) | 0.714 | **0.853** | 0.733 | **15.8** | 0 | Every variant made beats longer and specific with no invented names, but beat validation and coverage fell beyond the current prompt's spread, so no prompt change shipped. Causes from the failed beats: a two-actor example was copied; "to whom" moved the person acted on into `actors` (fixed in v3, which is why attribution rose); "and how it turned out" made the model fold the GM's set-up and outcome into the player's action and cite the GM's lines instead of the player's, which fails validation and merges scene beats away (74 → 52 verified). Shipped on the branch: an eval detail report (`evals/detail.py`, `--detail`): words per beat and names grounded in cited lines / elsewhere / invented. Next: one more round, v5 = v4 without the outcome clause.

rbrooks referenced this issue from a commit

2026-10-14 05:17:00 +00:00

evals: record the v5 richer-beat prompt runs; not shipped (#1189)

rbrooks referenced this issue from a commit

2026-10-14 05:49:26 +00:00

beats: credit an actor from inside the cited exchange, behind a switch that is off (#1189)

claude-bot commented

2026-10-14 05:50:23 +00:00

Author

Collaborator

Third round: v4 + a looser transcript check (branch fix/1189-v4-looser-check, commit 32076db4, switch CREDIT_ACTOR_INSIDE_CITED_SPAN, default off).

Rule. An actor who did not speak a cited line is accepted only when the beat cites at least two real lines no more than 120 s apart, the actor spoke a line between them, and nobody else did (every line in the span is the GM's or one of the beat's own actors). The actor's lines are added to the evidence and the repair recorded. No window outside the cited span: in misattributed_actor the line just after the outcome is a bystander's, and a window would credit them.

prompt (runs)	coverage	attribution	chronology	beat validation	precision	words/beat	verified beats
current (4)	0.751	0.515	0.905	0.840	1.000	10.9	74
v4 (3)	0.714	0.853	0.873	0.733	1.000	15.8	52
v4 + looser (3)	0.895	0.798	0.849	0.928	1.000	15.9	59.7

8 beats rescued per run, all read against the fixtures: none credits the wrong person. Still rejected: single-citation beats. Chronology of the written summary falls with v4 (beat order is still enforced in code). 13 new unit tests; 898 backend tests pass with the switch off, the CI stub eval is identical with it on or off.

v5 (v4 without the outcome clause) was dropped: on the real Session 14 it lost 21 of 76 verified beats.

Real-session comparison (read-only on prod): v4 kept 71 of 76 verified beats and told 61 as story against 34; v4 + looser is being added. Decision is the owner's.

Third round: v4 + a looser transcript check (branch `fix/1189-v4-looser-check`, commit 32076db4, switch `CREDIT_ACTOR_INSIDE_CITED_SPAN`, default off). **Rule.** An actor who did not speak a cited line is accepted only when the beat cites at least two real lines no more than 120 s apart, the actor spoke a line between them, and nobody else did (every line in the span is the GM's or one of the beat's own actors). The actor's lines are added to the evidence and the repair recorded. No window outside the cited span: in `misattributed_actor` the line just after the outcome is a bystander's, and a window would credit them. | prompt (runs) | coverage | attribution | chronology | beat validation | precision | words/beat | invented | verified beats | |---|---|---|---|---|---|---|---|---| | current (4) | 0.751 | 0.515 | 0.905 | 0.840 | 1.000 | 10.9 | 0 | 74 | | v4 (3) | 0.714 | 0.853 | 0.873 | 0.733 | 1.000 | 15.8 | 0 | 52 | | **v4 + looser (3)** | **0.895** | 0.798 | 0.849 | **0.928** | 1.000 | 15.9 | 0 | 59.7 | 8 beats rescued per run, all read against the fixtures: none credits the wrong person. Still rejected: single-citation beats. Chronology of the written summary falls with v4 (beat order is still enforced in code). 13 new unit tests; 898 backend tests pass with the switch off, the CI stub eval is identical with it on or off. v5 (v4 without the outcome clause) was dropped: on the real Session 14 it lost 21 of 76 verified beats. Real-session comparison (read-only on prod): v4 kept 71 of 76 verified beats and told 61 as story against 34; v4 + looser is being added. Decision is the owner's.

claude-bot commented

2026-10-14 06:22:33 +00:00

Author

Collaborator

Real-session finding (read-only on prod, Session 14): the largest rejection class on every prompt is "'GM' did not speak any cited line", and it is not a label mismatch (the GM's lines are labelled GM and gm_labels matches). Looking up the cited stamps:

A stamp inside or at the end of a long GM range line. E.g. a GM line [00:16:41-00:17:03]; the model cites 00:17:03, where a player's "Ok." also starts. build_transcript_index keys lines by their start stamp only, so the check sees the player. 00:53:24 lies inside a GM line [00:53:03-00:53:26] and is even reported "not in the transcript". The prompt already asks for a range line's start stamp; the model often gives the end or a moment inside.
The player's reply just after the GM. The GM says "Plyope Circus" at 00:47:58; the model cites 00:47:59 (a player's line). Similar for a City Watch description.

Owner decision (14 Oct): build (a) a cited stamp that falls inside a line's [start-end] span resolves to that line (as a range should be read; on by default, no misattribution possible since the moment is inside that speech), and (b) a GM-only look-back (an event crediting the GM that cites a player's line is accepted when the GM spoke within a few seconds before, and that GM line is added to the evidence), behind a switch, default off, decided after measurement. Then re-run the read-only Session 14 comparison with the current prompt and v4.

Real-session finding (read-only on prod, Session 14): the largest rejection class on every prompt is "'GM' did not speak any cited line", and it is not a label mismatch (the GM's lines are labelled `GM` and `gm_labels` matches). Looking up the cited stamps: 1. **A stamp inside or at the end of a long GM range line.** E.g. a GM line `[00:16:41-00:17:03]`; the model cites `00:17:03`, where a player's "Ok." also starts. `build_transcript_index` keys lines by their start stamp only, so the check sees the player. `00:53:24` lies inside a GM line `[00:53:03-00:53:26]` and is even reported "not in the transcript". The prompt already asks for a range line's start stamp; the model often gives the end or a moment inside. 2. **The player's reply just after the GM.** The GM says "Plyope Circus" at `00:47:58`; the model cites `00:47:59` (a player's line). Similar for a City Watch description. Owner decision (14 Oct): build **(a)** a cited stamp that falls inside a line's `[start-end]` span resolves to that line (as a range should be read; on by default, no misattribution possible since the moment is inside that speech), and **(b)** a GM-only look-back (an event crediting the GM that cites a player's line is accepted when the GM spoke within a few seconds before, and that GM line is added to the evidence), behind a switch, default off, decided after measurement. Then re-run the read-only Session 14 comparison with the current prompt and v4.

rbrooks referenced this issue from a commit

2026-10-14 06:55:25 +00:00

evals: report how specific the beats are; richer-beat prompt not shipped (#1189)

rbrooks referenced this issue from a commit

2026-10-14 06:55:25 +00:00

beats: a cited stamp inside a range line resolves to that line; GM look-back behind a switch that is off (#1189)

rbrooks referenced this issue from a commit

2026-10-14 06:55:25 +00:00

evals: record the #1189 cited-stamp-in-range runs, with and without the GM look-back

rbrooks referenced this issue from a commit

2026-10-14 15:03:06 +00:00

evals: report how specific the beats are; richer-beat prompt not shipped (#1189)

rbrooks referenced this issue from a commit

2026-10-14 15:03:06 +00:00

evals: record the v5 richer-beat prompt runs; not shipped (#1189)

rbrooks referenced this issue from a commit

2026-10-14 15:03:06 +00:00

beats: credit an actor from inside the cited exchange, behind a switch that is off (#1189)

rbrooks referenced this issue from a commit

2026-10-14 15:03:06 +00:00

beats: richer events, and three checks that stop dropping real ones (#1189)

rbrooks referenced this issue from a commit

2026-10-14 15:03:06 +00:00

changelog: the detail report fragment no longer says nothing changes (#1189)

rbrooks referenced this issue from a commit

2026-10-14 15:03:06 +00:00

tests: the range test pins (a) alone; the GM neighbour credits the line inside the range (#1189)

claude-bot referenced this issue from a pull request that will close it,

2026-10-14 15:03:16 +00:00

Richer session events, and three checks that stop dropping real ones (#1189) #1192

claude-bot commented

2026-10-14 15:03:22 +00:00

Author

Collaborator

The owner chose the v4 prompt with the looser checks, after reading the side-by-side on the real session. The comparison page is private, because it holds real session content.

One point on the decision: the looser span check itself rescued 0 events on that session. The "v4 + looser check" summary the owner preferred over plain v4 came from a separate v4 extraction, and extraction is not deterministic. So the choice between those two rests on sampling, and the case for the looser check rests on the synthetic fixtures.

PR #1192 ships:

the v4 prompt;
(a), a stamp inside a range line resolving to that line;
the span credit, switched on;
the GM's neighbouring line at 5 s, now checked both before and after the cited line, GM only, with nobody in between.

Before merging the PR on green: 921 summarisation tests passed and the CI stub eval cleared every threshold.

Session 14 itself has not been re-summarised on prod. That waits for the owner's say after 4.5.2 is deployed.

The owner chose the **v4 prompt with the looser checks**, after reading the side-by-side on the real session. The comparison page is private, because it holds real session content. One point on the decision: the looser span check itself rescued 0 events on that session. The "v4 + looser check" summary the owner preferred over plain v4 came from a separate v4 extraction, and extraction is not deterministic. So the choice between those two rests on sampling, and the case for the looser check rests on the synthetic fixtures. PR #1192 ships: - the v4 prompt; - (a), a stamp inside a range line resolving to that line; - the span credit, switched on; - the GM's neighbouring line at 5 s, now checked both before and after the cited line, GM only, with nobody in between. Before merging the PR on green: 921 summarisation tests passed and the CI stub eval cleared every threshold. Session 14 itself has not been re-summarised on prod. That waits for the owner's say after 4.5.2 is deployed.

claude-bot closed this issue

2026-10-14 15:33:23 +00:00

claude-bot referenced this issue from a commit

2026-10-14 15:33:23 +00:00

Merge PR #1192: richer session events, and three checks that stop dropping real ones (#1189)

claude-bot referenced this issue

2026-10-14 16:55:37 +00:00

chore(release): v4.5.2-rc1 #1193

Rows
Columns

Session events are too terse: the extraction prompt asks for one bare sentence #1189