#319 - [Review] Full audit of the session lifecycle: prep, recording, transcription, summarization, lore — scoping v4.0.0 - rbrooks/Quest-Board

claude-bot commented

2026-08-25 19:18:25 +00:00

Contributor

Tracking issue for a full review of everything touching the session lifecycle, ahead of a re-scoped v4.0.0. The FoundryVTT pillar currently sitting on v4.0.0–v4.3.0 moves to v5.x.

Why

Quest Board is being prepared as a hosted product. The core loop — scheduling, session planning, recording, summarization, lore management — has to be bulletproof without hands-on work from the operator or the customer. Two things forced the review:

The 2026-08-11 session summary mixed up the chronology of events and misattributed actions to the wrong player.
Four bugs (#302, #303, #304, #305) were filed after a single real session-prep run, one of them with the verdict "pretty useless".

Scope

Session prep / GM Workbench
Live session running and recording capture
Transcription and per-speaker attribution
Summarization and the whole AI pipeline (highlights, lore extraction, wiki proposals)
Post-session review, approval and lore management
Data durability across all of the above ("data is never lost")
Feature discoverability and workflow coherence
Hosted-tier ASR/LLM stack: managed APIs vs owned hardware, costed

Backlog issues to be pulled into the new v4.0.0: #121, #122, #123, #124, #125, #126, #128.

Headline finding so far — recording capture has no session clock

Confirmed against production data. Per-speaker WAV tracks are not on a shared timeline. Each track is that speaker's own speech, concatenated with the silence between their utterances deleted, so a track's duration equals that speaker's total talk time rather than the session's wall-clock duration. Whisper timestamps are therefore track-relative, and merge_attributed_transcript sorts values from six mutually incomparable clocks onto one axis.

Evidence — session 56cc4dee-99c0-470a-bd90-18e7621b4194 (2026-08-12 01:00–04:00 UTC), stored transcript, 1904 lines, 0 non-monotonic, 0 unparsed:

Speaker	Lines	First	Last
DesertCreosote	675	00:00:04	00:54:47
Viquilonto (Viq)	203	00:00:03	00:15:46
Wyatt	327	00:00:01	00:14:59
Idani	343	00:00:00	00:14:45
Harrowhark	223	00:00:00	00:14:11
Clio	133	00:00:00	00:10:07

Max timestamp in the entire transcript is 00:54:47 for a multi-hour session, and the per-speaker durations sum to ~2.08 h — approximately the real session length, which is the signature of speech-only concatenation.

Mechanism, in bot/questboard_bot/cogs/recording.py:

_session_bytes is documented (line 118) as "Bytes on the first speaker's timeline" and advances only for packets whose uid == next(iter(self._writers)) (lines 183-184, 204-205) — one arbitrary user, whose own track only advances while they transmit.
Late speakers are padded once, on their first packet, to that value (lines 172-174).
The library's jitter-buffer PLC frames, which exist to maintain continuity through silence, are explicitly discarded (lines 156-161). The pinned fork ships a SilenceGeneratorSink that would fill gaps in real time; the bot never uses it.
Commit 7ddff16 ("Fix per-speaker timestamp alignment", Apr 2026) claims to track wall-clock position. It does not — that is the original misdiagnosis, and test_recording.py:534-548 locks the broken model in.

This affects every multi-speaker recording Quest Board has ever produced, not just the 08-11 session.

Status

Review in progress. A written report will land in docs/.internal/ first; issues get filed against the new v4.0.0 milestone once the findings are agreed.

Tracking issue for a full review of everything touching the session lifecycle, ahead of a re-scoped **v4.0.0**. The FoundryVTT pillar currently sitting on v4.0.0–v4.3.0 moves to **v5.x**. ## Why Quest Board is being prepared as a **hosted product**. The core loop — scheduling, session planning, recording, summarization, lore management — has to be bulletproof without hands-on work from the operator or the customer. Two things forced the review: 1. The 2026-08-11 session summary **mixed up the chronology** of events and **misattributed actions to the wrong player**. 2. Four bugs (#302, #303, #304, #305) were filed after a single real session-prep run, one of them with the verdict "pretty useless". ## Scope - Session prep / GM Workbench - Live session running and recording capture - Transcription and per-speaker attribution - Summarization and the whole AI pipeline (highlights, lore extraction, wiki proposals) - Post-session review, approval and lore management - Data durability across all of the above ("data is never lost") - Feature discoverability and workflow coherence - Hosted-tier ASR/LLM stack: managed APIs vs owned hardware, costed Backlog issues to be pulled into the new v4.0.0: #121, #122, #123, #124, #125, #126, #128. ## Headline finding so far — recording capture has no session clock **Confirmed against production data.** Per-speaker WAV tracks are not on a shared timeline. Each track is that speaker's own speech, concatenated with the silence between their utterances deleted, so a track's duration equals that speaker's total talk time rather than the session's wall-clock duration. Whisper timestamps are therefore track-relative, and `merge_attributed_transcript` sorts values from six mutually incomparable clocks onto one axis. Evidence — session `56cc4dee-99c0-470a-bd90-18e7621b4194` (2026-08-12 01:00–04:00 UTC), stored transcript, 1904 lines, 0 non-monotonic, 0 unparsed: | Speaker | Lines | First | Last | |---|---|---|---| | DesertCreosote | 675 | 00:00:04 | **00:54:47** | | Viquilonto (Viq) | 203 | 00:00:03 | 00:15:46 | | Wyatt | 327 | 00:00:01 | 00:14:59 | | Idani | 343 | 00:00:00 | 00:14:45 | | Harrowhark | 223 | 00:00:00 | 00:14:11 | | Clio | 133 | 00:00:00 | 00:10:07 | Max timestamp in the entire transcript is **00:54:47** for a multi-hour session, and the per-speaker durations sum to ~2.08 h — approximately the real session length, which is the signature of speech-only concatenation. Mechanism, in `bot/questboard_bot/cogs/recording.py`: - `_session_bytes` is documented (line 118) as "Bytes on the first speaker's timeline" and advances only for packets whose `uid == next(iter(self._writers))` (lines 183-184, 204-205) — one arbitrary user, whose own track only advances while they transmit. - Late speakers are padded once, on their first packet, to that value (lines 172-174). - The library's jitter-buffer PLC frames, which exist to maintain continuity through silence, are explicitly discarded (lines 156-161). The pinned fork ships a `SilenceGeneratorSink` that would fill gaps in real time; the bot never uses it. - Commit `7ddff16` ("Fix per-speaker timestamp alignment", Apr 2026) claims to track wall-clock position. It does not — that is the original misdiagnosis, and `test_recording.py:534-548` locks the broken model in. This affects **every multi-speaker recording Quest Board has ever produced**, not just the 08-11 session. ## Status Review in progress. A written report will land in `docs/.internal/` first; issues get filed against the new v4.0.0 milestone once the findings are agreed.

claude-bot commented

2026-08-25 19:26:02 +00:00

Author

Contributor

Correction to the issue body, plus the recommended fix

Deeper analysis against the pinned library source corrected two claims in the body above. The root cause and the production evidence are unchanged — the corrections are about mechanism, and they matter because one of them would send someone down a dead end.

Corrections

PLC / FakePackets were never a timeline mechanism. The body says the discarded jitter-buffer PLC frames "exist to maintain continuity through silence". That is wrong. PLC fires only on sequence-number gaps (library opus.py:162-182), and sequence numbers pause during silence — so inter-spurt gaps generate zero FakePackets. Re-enabling them cannot fix this bug. There is also a documented reason the skip exists: commit 02898b3 records DAVE-deferred packets causing a 148× PLC explosion (340 MB for 12 s of audio). Do not "fix" this by accepting FakePackets, and disregard the body's suggestion that wiring up SilenceGeneratorSink is the answer.
Line numbers and the reference user. The FakePacket skip is at recording.py:163-166, not 156-161. And the reference user is the first user whose packet reached the sink — the user is None drop (161-162) and DTX-first packets can each make that someone other than the first person to speak.

One further note on evidence: the transcript's zero non-monotonic timestamps proves nothing, since the backend sorts by construction. The load-bearing evidence is the max track end of 00:54:47 against a multi-hour session, plus impossible sustained speech density (Idani at one line per 2.6 s for their whole track).

Recommended fix — wall-clock anchoring in the sink

Give PerUserPCMSink a t0 (the same time.monotonic() as rec.started_at, with an injectable clock for tests). Per packet:

target_frames = int((clock() - t0) * 48000)
if target_frames - written_frames > 250ms worth:
    pad that user's file with silence up to target (4-byte aligned, bounded chunks)
append the decoded PCM

At close(), tail-pad every track to duration_s. Delete _session_bytes, the reference-uid checks, and the first-packet pad — the gap logic subsumes all three. Placement error is bounded by jitter-buffer delay + 250 ms per spurt and is non-cumulative over a 4-hour session. Loss, mutes, leave/rejoin and reconnects all degrade to silence automatically.

Two required companions:

Switch raw capture from wave to headerless .s16le, converting with ffmpeg -f s16le -ar 48000 -ac 2 -i. Gap-filled 48 kHz stereo hits WAV's 4 GB RIFF cap at ~6 h (the current default cap), and headerless also allows sparse-seek gaps.
Reset the per-uid Opus decoder on SSRC change (recording.py:170) — a stale decoder garbles audio after a rejoin.

Deferred: RTP-timestamp anchoring

The sink does receive .timestamp / .ssrc / .sequence (library opus.py:33-45), and per-SSRC wall-anchored wrapped-diff placement would be frame-accurate. But it assumes every client advances its timestamp across silence — and if any client does not, it silently recreates this exact bug. Ship wall-clock anchoring, log ts_delta_vs_wall_delta in AUDIO_DIAG for one release, and adopt RTP anchoring only if the data justifies it.

Regression test (pure unit, no Discord needed)

Fake-clock-injected sink; spurts at t=0–1 s and t=60–61 s for user A, t=120 s for user B; close() at t=300 s. Assert both WAVs are 300 s ±300 ms with speech bytes at exactly the right byte offsets. This fails today in three independent ways, and it extends the existing _FakeDecoder pattern so it runs in CI as-is. Note that test_recording.py:534-548 currently asserts the broken model and must be rewritten.

Tripwires so this can never regress silently

close() logs the expected/written ratio per user — one grep proves the fix on the first real session
Backend fails processing if any WAV's duration is < 0.9 × duration_seconds, before spending a GPU run (process_audio already holds duration_seconds at reminder_tasks.py:1987-1994 and ignores it)
Backend fails pre-merge if the max segment end is < 0.6 × duration_seconds
The live dashboard's existing seconds_captured vs elapsed_seconds becomes a real-time post-deploy pass/fail

## Correction to the issue body, plus the recommended fix Deeper analysis against the pinned library source corrected two claims in the body above. The root cause and the production evidence are unchanged — the corrections are about *mechanism*, and they matter because one of them would send someone down a dead end. ### Corrections 1. **PLC / FakePackets were never a timeline mechanism.** The body says the discarded jitter-buffer PLC frames "exist to maintain continuity through silence". That is wrong. PLC fires only on *sequence-number* gaps (library `opus.py:162-182`), and sequence numbers pause during silence — so inter-spurt gaps generate zero FakePackets. Re-enabling them cannot fix this bug. There is also a documented reason the skip exists: commit `02898b3` records DAVE-deferred packets causing a 148× PLC explosion (340 MB for 12 s of audio). **Do not "fix" this by accepting FakePackets**, and disregard the body's suggestion that wiring up `SilenceGeneratorSink` is the answer. 2. **Line numbers and the reference user.** The FakePacket skip is at `recording.py:163-166`, not 156-161. And the reference user is the first user whose packet *reached the sink* — the `user is None` drop (161-162) and DTX-first packets can each make that someone other than the first person to speak. One further note on evidence: the transcript's zero non-monotonic timestamps proves nothing, since the backend sorts by construction. The load-bearing evidence is the max track end of 00:54:47 against a multi-hour session, plus impossible sustained speech density (Idani at one line per 2.6 s for their whole track). ### Recommended fix — wall-clock anchoring in the sink Give `PerUserPCMSink` a `t0` (the same `time.monotonic()` as `rec.started_at`, with an injectable clock for tests). Per packet: ``` target_frames = int((clock() - t0) * 48000) if target_frames - written_frames > 250ms worth: pad that user's file with silence up to target (4-byte aligned, bounded chunks) append the decoded PCM ``` At `close()`, tail-pad every track to `duration_s`. Delete `_session_bytes`, the reference-uid checks, and the first-packet pad — the gap logic subsumes all three. Placement error is bounded by jitter-buffer delay + 250 ms per spurt and is **non-cumulative** over a 4-hour session. Loss, mutes, leave/rejoin and reconnects all degrade to silence automatically. Two required companions: - **Switch raw capture from `wave` to headerless `.s16le`**, converting with `ffmpeg -f s16le -ar 48000 -ac 2 -i`. Gap-filled 48 kHz stereo hits WAV's 4 GB RIFF cap at ~6 h (the current default cap), and headerless also allows sparse-seek gaps. - **Reset the per-uid Opus decoder on SSRC change** (`recording.py:170`) — a stale decoder garbles audio after a rejoin. ### Deferred: RTP-timestamp anchoring The sink does receive `.timestamp` / `.ssrc` / `.sequence` (library `opus.py:33-45`), and per-SSRC wall-anchored wrapped-diff placement would be frame-accurate. But it assumes every client advances its timestamp across silence — and if any client does not, it silently recreates this exact bug. Ship wall-clock anchoring, log `ts_delta_vs_wall_delta` in `AUDIO_DIAG` for one release, and adopt RTP anchoring only if the data justifies it. ### Regression test (pure unit, no Discord needed) Fake-clock-injected sink; spurts at t=0–1 s and t=60–61 s for user A, t=120 s for user B; `close()` at t=300 s. Assert both WAVs are 300 s ±300 ms with speech bytes at exactly the right byte offsets. This fails today in three independent ways, and it extends the existing `_FakeDecoder` pattern so it runs in CI as-is. Note that `test_recording.py:534-548` currently asserts the broken model and must be rewritten. ### Tripwires so this can never regress silently - `close()` logs the expected/written ratio per user — one grep proves the fix on the first real session - Backend fails processing if any WAV's duration is < 0.9 × `duration_seconds`, before spending a GPU run (`process_audio` already holds `duration_seconds` at `reminder_tasks.py:1987-1994` and ignores it) - Backend fails pre-merge if the max segment end is < 0.6 × `duration_seconds` - The live dashboard's existing `seconds_captured` vs `elapsed_seconds` becomes a real-time post-deploy pass/fail

claude-bot commented

2026-08-25 19:32:47 +00:00

Author

Contributor

Review complete — report landed

Written up at docs/.internal/session-lifecycle-review-2026-08.md, with the six full audits (≈2,550 lines, file:line evidence throughout) in docs/.internal/session-review-2026-08/.

No issues filed yet — scope gets agreed first, then they go up against a new v4.0.0 milestone.

Headline additions since the last comment

VAD is off in production. There is no vad_config row, so VAD_DEFAULT_ENABLED = False applies. That creates a hard sequencing constraint: fixing capture makes every track run the full wall-clock length, taking ASR input from ~2 track-hours to ~17.5, and re-arming Whisper's hallucination-on-silence failure, which is dormant today only because the silence was being deleted. The capture fix and a silence strategy must ship together, or the fix is a regression. The current -30 dB noise floor is also too aggressive for Discord per-user tracks (which are digitally silent at −∞ dB when nobody transmits) and will cut quiet speech — validate against a real recording before enabling.

Counts: 2 critical + 2 high in capture, 2 high in transcription/merge, 2 critical + 2 high in the AI pipeline, 3 critical + 4 high in data durability, 8 major UX findings plus ~30 silent-failure sites.

Three data-loss criticals, none related to the timeline bug:

Re-submitting audio has no state guard — it destroys GM-edited transcript, summary and curated highlights, and the bot's fixed per-session filenames mean a "part 2" recording overwrites part 1's WAVs on disk, unrecoverably.
No task_acks_late / task_reject_on_worker_lost anywhere: a routine docker compose up -d --build mid-transcription strands the session at processing forever, with both retry endpoints 409-ing. SQL-only escape. Eleven such stuck states were enumerated.
The bot deletes recordings it told the GM were saved (no HANDOFF_MARKER on backend-unreachable, no retry, swept after 60 min).

Privacy: audio_service.py:524-528 logs str(data)[:500] at INFO — that repr opens with segment text and speaker, in direct violation of the module docstring at line 24. Verified on prod: 1 line in retained worker logs.

Hosted economics: AssemblyAI Universal async + Claude Sonnet 5, VAD mandatory — ≈$0.95/session, ≈$4.11/customer-month at weekly play. Break-even vs owned hardware is ~60–75 groups once your time and redundancy are priced honestly. Price per group, not per seat.

Proposed tiers

Tier 0 (v3.11.5 hotfix, this week): wall-clock anchoring + .s16le container + SSRC decoder reset; enable VAD at a corrected floor; the two duration-invariant guards; remove the PII log line; rewrite test_recording.py:534-548 (it currently asserts the broken model) and add the wall-clock regression test.
Tier 1 (v4.0.0): accuracy re-architecture (deterministic speaker→character resolution, validated beat extraction, segment rows), the durability set, the product-surface set, prep rebuilt around the session, plus backlog #121–#126 and #128 and bugs #302–#306, #281, #289, #294, #175, #176, #182.
Tier 2 (v5.0.0): the FoundryVTT pillar, renumbered from v4.0.0–v4.3.0.

Five open decisions are listed in §8 of the report.

## Review complete — report landed Written up at **`docs/.internal/session-lifecycle-review-2026-08.md`**, with the six full audits (≈2,550 lines, `file:line` evidence throughout) in `docs/.internal/session-review-2026-08/`. No issues filed yet — scope gets agreed first, then they go up against a new v4.0.0 milestone. ### Headline additions since the last comment **VAD is off in production.** There is no `vad_config` row, so `VAD_DEFAULT_ENABLED = False` applies. That creates a hard sequencing constraint: fixing capture makes every track run the full wall-clock length, taking ASR input from ~2 track-hours to ~17.5, and **re-arming Whisper's hallucination-on-silence failure**, which is dormant today only because the silence was being deleted. **The capture fix and a silence strategy must ship together**, or the fix is a regression. The current `-30 dB` noise floor is also too aggressive for Discord per-user tracks (which are digitally silent at −∞ dB when nobody transmits) and will cut quiet speech — validate against a real recording before enabling. **Counts:** 2 critical + 2 high in capture, 2 high in transcription/merge, 2 critical + 2 high in the AI pipeline, 3 critical + 4 high in data durability, 8 major UX findings plus ~30 silent-failure sites. **Three data-loss criticals**, none related to the timeline bug: - Re-submitting audio has no state guard — it destroys GM-edited transcript, summary and curated highlights, and the bot's fixed per-session filenames mean a "part 2" recording **overwrites part 1's WAVs on disk**, unrecoverably. - No `task_acks_late` / `task_reject_on_worker_lost` anywhere: a routine `docker compose up -d --build` mid-transcription strands the session at `processing` forever, with both retry endpoints 409-ing. SQL-only escape. Eleven such stuck states were enumerated. - The bot deletes recordings it told the GM were saved (no `HANDOFF_MARKER` on backend-unreachable, no retry, swept after 60 min). **Privacy:** `audio_service.py:524-528` logs `str(data)[:500]` at INFO — that repr opens with segment `text` and `speaker`, in direct violation of the module docstring at line 24. Verified on prod: 1 line in retained worker logs. **Hosted economics:** AssemblyAI Universal async + Claude Sonnet 5, VAD mandatory — ≈$0.95/session, ≈$4.11/customer-month at weekly play. Break-even vs owned hardware is ~60–75 groups once your time and redundancy are priced honestly. Price per group, not per seat. ### Proposed tiers - **Tier 0 (v3.11.5 hotfix, this week):** wall-clock anchoring + `.s16le` container + SSRC decoder reset; enable VAD at a corrected floor; the two duration-invariant guards; remove the PII log line; rewrite `test_recording.py:534-548` (it currently asserts the broken model) and add the wall-clock regression test. - **Tier 1 (v4.0.0):** accuracy re-architecture (deterministic speaker→character resolution, validated beat extraction, segment rows), the durability set, the product-surface set, prep rebuilt around the session, plus backlog #121–#126 and #128 and bugs #302–#306, #281, #289, #294, #175, #176, #182. - **Tier 2 (v5.0.0):** the FoundryVTT pillar, renumbered from v4.0.0–v4.3.0. Five open decisions are listed in §8 of the report.

Was	Now
v4.0.0 — FoundryVTT Phase 1: Foundation	v5.0.0
v4.1.0 — FoundryVTT Phase 2: NPC Push	v5.1.0
v4.2.0 — FoundryVTT Phase 2.5: Live Scene Push	v5.2.0
v4.3.0 — FoundryVTT Phase 3: Lore Sync & Deeper Pulls	v5.3.0
v4.4.0 — Notification Controls	v5.4.0

Milestone	New	Moved in	Total
v3.11.5 — Recording Clock Hotfix	9 (#320–#328)	—	9
v4.0.0 — Session Pipeline Accuracy	21 (#329–#349)	#281, #289, #294, #232	25
v4.1.0 — Data Durability & Recovery	20 (#397–#416)	#306, #175, #176	23
v4.2.0 — AI Provider Abstraction & BYO-AI	11 (#350–#360)	#128	12
v4.3.0 — Product Surface	29 (#368–#396)	#302–#305	33
v4.4.0 — UX Overhaul	7 (#361–#367)	#182	8
v4.5.0 — Table & Safety Tools	—	#121–#126	6

Rows
Columns

[Review] Full audit of the session lifecycle: prep, recording, transcription, summarization, lore — scoping v4.0.0 #319

Why

Scope

Headline finding so far — recording capture has no session clock

Status

Correction to the issue body, plus the recommended fix

Corrections

Recommended fix — wall-clock anchoring in the sink

Deferred: RTP-timestamp anchoring

Regression test (pure unit, no Discord needed)

Tripwires so this can never regress silently

Review complete — report landed

Headline additions since the last comment

Proposed tiers

Milestones restructured and issues filed

Milestone renumbering

The new line

Decisions encoded in the scope

Not filed, deliberately