Wiki facts suggested by the local model after a session (Status first), for GM approval #1280

Closed
opened 2026-10-29 17:32:21 +00:00 by claude-bot · 6 comments
Collaborator

Split out of #1279 at the owner's request (2026-10-29), so stage 2 of the fact automation isn't forgotten. #1279 builds the infobox and the facts that need no AI (First met / Last seen, Player and pronouns, sheet fields, content-pack fields, Region from the map).

Goal: after a session is processed, suggest updates to the facts of the pages that appeared in it, as suggestions a GM approves. Start with Status (alive / dead / missing / unknown), then a short list of fields per page type that the owner chooses (e.g. Leader, Ruler, Owner, Allegiance).

Approach, sized for the self-hosted open-weights model

  • One narrow question per page that appeared (lore_session_appearances), over the transcript excerpts that mention it. The answer is constrained to a JSON schema (the endpoint honours json_schema), choosing from fixed values where the field has them.
  • Every suggested change carries a quote, checked against the transcript the way summary beats are validated. A suggestion whose quote isn't found is dropped, and the count is recorded on the run.
  • Through the existing draft review (lore_entry_drafts.current_sidebar_fields): GM-only until approved, and it never overwrites a fact the GM wrote without the GM saying so.
  • Roughly one short call per page mentioned per session, run in the worker after summarisation.

Not in scope: open-ended inference (a settlement's population, an implied motive) and synthesis across many sessions; both are unreliable on the local model.

Measure before widening: run it over the owner's real sessions and record how many suggestions were accepted, edited or rejected per field, before adding more fields.

Depends on #1279.

Split out of #1279 at the owner's request (2026-10-29), so stage 2 of the fact automation isn't forgotten. #1279 builds the infobox and the facts that need no AI (First met / Last seen, Player and pronouns, sheet fields, content-pack fields, Region from the map). **Goal:** after a session is processed, suggest updates to the facts of the pages that appeared in it, as suggestions a GM approves. Start with **Status** (alive / dead / missing / unknown), then a short list of fields per page type that the owner chooses (e.g. Leader, Ruler, Owner, Allegiance). **Approach, sized for the self-hosted open-weights model** - **One narrow question per page that appeared** (`lore_session_appearances`), over the transcript excerpts that mention it. The answer is constrained to a JSON schema (the endpoint honours `json_schema`), choosing from fixed values where the field has them. - **Every suggested change carries a quote**, checked against the transcript the way summary beats are validated. A suggestion whose quote isn't found is dropped, and the count is recorded on the run. - **Through the existing draft review** (`lore_entry_drafts.current_sidebar_fields`): GM-only until approved, and it never overwrites a fact the GM wrote without the GM saying so. - Roughly one short call per page mentioned per session, run in the worker after summarisation. **Not in scope:** open-ended inference (a settlement's population, an implied motive) and synthesis across many sessions; both are unreliable on the local model. **Measure before widening:** run it over the owner's real sessions and record how many suggestions were accepted, edited or rejected per field, before adding more fields. Depends on #1279.
Author
Collaborator

Picking this up now that #1279 has merged (#1324). First a drawing (lane 6j) of where a suggested fact change appears and how a GM approves, edits or rejects it, with its quote. The questions go to the owner before anything is built.

Picking this up now that #1279 has merged (#1324). First a drawing (lane 6j) of where a suggested fact change appears and how a GM approves, edits or rejects it, with its quote. The questions go to the owner before anything is built.
Author
Collaborator

The owner's answers (2026-11-03), recorded as D323–D326 (commit ae62f6fd on design/1280-fact-suggestions; the drawing is lane 6j):

  • Q214 → D323: start with five facts: Status (Alive / Dead / Missing / Unknown) on NPC, PC, Creature and Deity pages; Allegiances on NPC; Ruler on Settlement and Country; Leader on Organization; Owner on Business.
  • Q215 → D324: a used suggestion keeps the fact's own "Who sees it". A fact that was empty becomes visible to everyone at the table.
  • Q216 → D325: after the release, run once over past sessions, newest first, on prod's own worker. The number of suggestions used, changed or turned down is the measure for widening the list.
  • Q217 → D326: suggestions show in both Suggested wiki updates and the page editor.

Build: branch feat/1280-fact-suggestions, with the backend and screen built in parallel from one spec.

  • Storage: a fact_suggestions table rather than the draft review the issue first proposed. Each change needs its own Use it, Change it, Not this one and Undo, and the use, change and rejection counts are what D325 measures.
  • Quotes: every quote is checked against the transcript segments. The speaker and the time come from the matching segment, never from the model.
**The owner's answers (2026-11-03), recorded as D323–D326** (commit ae62f6fd on `design/1280-fact-suggestions`; the drawing is lane 6j): - **Q214 → D323:** start with five facts: Status (Alive / Dead / Missing / Unknown) on NPC, PC, Creature and Deity pages; Allegiances on NPC; Ruler on Settlement and Country; Leader on Organization; Owner on Business. - **Q215 → D324:** a used suggestion keeps the fact's own "Who sees it". A fact that was empty becomes visible to everyone at the table. - **Q216 → D325:** after the release, run once over past sessions, newest first, on prod's own worker. The number of suggestions used, changed or turned down is the measure for widening the list. - **Q217 → D326:** suggestions show in both Suggested wiki updates and the page editor. **Build:** branch `feat/1280-fact-suggestions`, with the backend and screen built in parallel from one spec. - **Storage:** a `fact_suggestions` table rather than the draft review the issue first proposed. Each change needs its own Use it, Change it, Not this one and Undo, and the use, change and rejection counts are what D325 measures. - **Quotes:** every quote is checked against the transcript segments. The speaker and the time come from the matching segment, never from the model.
Author
Collaborator

Done: merged in #1325 (D323–D326, lane 6j).

  • The suggestion task: after summarisation, the worker asks the campaign's LLM one schema-constrained question per appearing page about Status, Allegiances, Ruler, Leader or Owner. It covers only the lines that mention the page.
    • Every change must quote the transcript. The speaker and the time come from the matched line, never from the model.
    • Dropped and same-value changes are counted on the run.
  • Using a suggestion: suggestions are GMs-only until used, in Suggested wiki updates (Use it / Change it… / Not this one / Undo) and in the page editor. A used fact keeps its own audience, and a new one is for everyone at the table. History records it.
  • Privacy: quotes follow the transcript.
    • Retention destroys them.
    • The bin hides them.
    • A member's erasure removes the suggestions quoting their lines (matched by line).
    • A reprocess forgets the old ones before renumbering the lines.
    • The backfill never reads a binned transcript.
    • Each of these is tested. The security review found three, and CI's transcript-bin guard found one more.
  • Checks: the render check was clean (44 shots, axe clean, players get 403). Owner wording part t is approved, and CI passed.

After the 4.7.0 deploy, on the owner's say (D325), run once:

celery -A app.tasks.celery_app call app.tasks.fact_tasks.suggest_fact_changes_backfill

Then measure used, changed and rejected per fact before widening the list.

**Done:** merged in #1325 (D323–D326, lane 6j). - **The suggestion task:** after summarisation, the worker asks the campaign's LLM one schema-constrained question per appearing page about Status, Allegiances, Ruler, Leader or Owner. It covers only the lines that mention the page. - Every change must quote the transcript. The speaker and the time come from the matched line, never from the model. - Dropped and same-value changes are counted on the run. - **Using a suggestion:** suggestions are GMs-only until used, in Suggested wiki updates (Use it / Change it… / Not this one / Undo) and in the page editor. A used fact keeps its own audience, and a new one is for everyone at the table. History records it. - **Privacy:** quotes follow the transcript. - Retention destroys them. - The bin hides them. - A member's erasure removes the suggestions quoting their lines (matched by line). - A reprocess forgets the old ones before renumbering the lines. - The backfill never reads a binned transcript. - Each of these is tested. The security review found three, and CI's transcript-bin guard found one more. - **Checks:** the render check was clean (44 shots, axe clean, players get 403). Owner wording part t is approved, and CI passed. **After the 4.7.0 deploy, on the owner's say** (D325), run once: ``` celery -A app.tasks.celery_app call app.tasks.fact_tasks.suggest_fact_changes_backfill ``` Then measure used, changed and rejected per fact before widening the list.
Author
Collaborator

Dev rehearsal of the past-sessions pass (4.7.0-rc2, 2026-11-04, owner's go-ahead).

  • The run: suggest_fact_changes_backfill on dev took 129 s: 44 sessions, all done, 0 skipped, 0 failed.
  • The counts: 176 pages asked; 8 suggested; 0 dropped for a missing quote; 0 the same as the stored value; 0 failed calls.
  • The suggestions: all from The Amberwake Accord and its GM copy, so 4 distinct: Leader ×2, Ruler, Owner. None was a Status change.
  • Quality: every quote was in the transcript, but three of the four didn't say the change, only mentioned the person near the page. Only "Orla Brennock is behind it" (Owner) supports its suggestion.

Improved in #1336 before prod's run: the prompt asks for the line that states the change, and a code check requires the quote to name the new value and hold a word for the fact. Unsupported suggestions are dropped and counted (dropped_unsupported). With that check, the dev set would keep 1 of the 4.

To compare on dev: an rc with #1336, then clear dev's 8 rc2 suggestions and their run counts, and rerun the pass.

**Dev rehearsal of the past-sessions pass (4.7.0-rc2, 2026-11-04, owner's go-ahead).** - **The run:** `suggest_fact_changes_backfill` on dev took 129 s: 44 sessions, all done, 0 skipped, 0 failed. - **The counts:** 176 pages asked; 8 suggested; 0 dropped for a missing quote; 0 the same as the stored value; 0 failed calls. - **The suggestions:** all from The Amberwake Accord and its GM copy, so 4 distinct: Leader ×2, Ruler, Owner. None was a Status change. - **Quality:** every quote was in the transcript, but three of the four didn't say the change, only mentioned the person near the page. Only "Orla Brennock is behind it" (Owner) supports its suggestion. **Improved in #1336** before prod's run: the prompt asks for the line that states the change, and a code check requires the quote to name the new value and hold a word for the fact. Unsupported suggestions are dropped and counted (`dropped_unsupported`). With that check, the dev set would keep 1 of the 4. **To compare on dev:** an rc with #1336, then clear dev's 8 rc2 suggestions and their run counts, and rerun the pass.
Author
Collaborator

Rerun on dev with 4.7.0-rc3 (#1336's quote check), 2026-11-05. I cleared rc2's 8 pending suggestions and the run counts, then ran the backfill again: 44 sessions, 176 pages asked.

  • 2 suggested, both The Drowned Lantern → Owner: Orla Brennock, quoting "Orla Brennock is behind it" (once per Amberwake campaign).
  • 4 dropped as unsupported: the three weak ones the rc2 run kept (Tamsin Reed as Ruler, Petronella Vay and Jory Callander as Leader). The model still offered them, and the code check dropped them.
  • 0 dropped for no quote, 0 the same as stored, 0 failed calls.

So the model alone still reaches for "a line that mentions the person". supports() is what keeps the queue honest, and the run's dropped_unsupported count shows how often it does so. That count is the number to watch on prod.

**Rerun on dev with 4.7.0-rc3 (#1336's quote check), 2026-11-05.** I cleared rc2's 8 pending suggestions and the run counts, then ran the backfill again: 44 sessions, 176 pages asked. - **2 suggested**, both The Drowned Lantern → Owner: Orla Brennock, quoting *"Orla Brennock is behind it"* (once per Amberwake campaign). - **4 dropped as unsupported**: the three weak ones the rc2 run kept (Tamsin Reed as Ruler, Petronella Vay and Jory Callander as Leader). The model still offered them, and the code check dropped them. - 0 dropped for no quote, 0 the same as stored, 0 failed calls. So the model alone still reaches for "a line that mentions the person". `supports()` is what keeps the queue honest, and the run's `dropped_unsupported` count shows how often it does so. That count is the number to watch on prod.
Author
Collaborator

Backfill on prod (2026-11-06, on the owner's say): task c5bcbbe8: 3 sessions done, 0 skipped, 0 failed. It made no suggestions and no AI calls (pages_asked: 0 on every run).

The cause is the data, not a fault. The pass asks only about wiki pages linked to the session (lore_session_appearances). On prod, "A Merry Band of Thieves" has 63 live pages but only one appearance link in the whole campaign, and none on sessions 13, 14 or 16, which are the three sessions with transcript lines. On dev, the pages had appearance links, which is why it found candidates there.

Possible follow-up, for the owner to decide: also ask about campaign pages whose title or alias is named in the transcript, even when no appearance link exists. excerpts_for already finds mentions, so most of this is a wider candidate query plus a cap on pages per session.

**Backfill on prod (2026-11-06, on the owner's say):** task `c5bcbbe8`: 3 sessions done, 0 skipped, 0 failed. It made **no suggestions and no AI calls** (`pages_asked: 0` on every run). The cause is the data, not a fault. The pass asks only about wiki pages linked to the session (`lore_session_appearances`). On prod, "A Merry Band of Thieves" has 63 live pages but only **one** appearance link in the whole campaign, and none on sessions 13, 14 or 16, which are the three sessions with transcript lines. On dev, the pages had appearance links, which is why it found candidates there. Possible follow-up, for the owner to decide: also ask about campaign pages whose title or alias is named in the transcript, even when no appearance link exists. `excerpts_for` already finds mentions, so most of this is a wider candidate query plus a cap on pages per session.
Sign in to join this conversation.
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/Quest-Board#1280
No description provided.