#320 - [Recording] Anchor per-speaker capture to a wall clock — tracks have never shared a session timeline - rbrooks/Quest-Board

claude-bot commented

2026-08-25 20:38:40 +00:00

Contributor

Severity: CRITICAL. Found in the August 2026 session lifecycle review (#319). This is the root cause of the scrambled chronology and misattributed actions reported after the 2026-08-11 session, and it affects every multi-speaker recording Quest Board has ever produced.

The defect

Per-speaker WAV tracks are not on a shared timeline. PerUserPCMSink.write() appends decoded PCM only when a packet for that user arrives, and Discord clients stop transmitting entirely during silence (roughly five DTX hangover frames, ~100 ms, then nothing). Every pause is therefore deleted from that speaker's track, cumulatively, for hours. A track's duration ends up equal to that speaker's total talk time, not the session's wall-clock length.

Whisper's timestamps are relative to the file it is given, so each track carries its own private, compressed clock. merge_attributed_transcript then sorts segments from six mutually incomparable clocks onto one axis.

The one piece of machinery that looks like a session clock is not one. _session_bytes is documented at recording.py:118 as "Bytes on the first speaker's timeline" and advances only for packets where uid == next(iter(self._writers)) — one arbitrary user, whose own track also only advances while they transmit. Late speakers are padded once, on their first packet, to that value.

Evidence

bot/questboard_bot/cogs/recording.py:156-215 — write() is the only place audio is produced. It never reads data.packet.timestamp or .sequence and never consults a wall clock.
bot/questboard_bot/cogs/recording.py:118-119 — the _session_bytes comment.
bot/questboard_bot/cogs/recording.py:172-174 — the one-time first-packet pad.
bot/questboard_bot/cogs/recording.py:183-184, :204-205 — _session_bytes advances only for the reference uid.
Commit 7ddff16 ("Fix per-speaker timestamp alignment", Apr 2026) introduced this and its message claims _session_bytes "tracks the wall-clock position of the first speaker". It has never read a clock. That commit is the original misdiagnosis.

Production confirmation

Session 56cc4dee-99c0-470a-bd90-18e7621b4194, ~3 hours scheduled. Stored transcript: 1904 lines, 0 unparsed.

Speaker	Lines	First	Last
DesertCreosote	675	00:00:04	00:54:47
Viquilonto (Viq)	203	00:00:03	00:15:46
Wyatt	327	00:00:01	00:14:59
Idani	343	00:00:00	00:14:45
Harrowhark	223	00:00:00	00:14:11
Clio	133	00:00:00	00:10:07

Max timestamp 00:54:47 for a multi-hour session, per-speaker durations summing to ~2.08 h, and impossible sustained speech density (Idani: one utterance every 2.6 seconds for their entire track).

Proposed fix — wall-clock anchoring

Give PerUserPCMSink a t0 (the same time.monotonic() sample as rec.started_at, with an injectable clock parameter for tests). Per packet, for user uid:

target_frames = int((clock() - t0) * 48000)
gap = target_frames - frames_written[uid]
if gap > GAP_THRESHOLD (250 ms worth):
    write `gap` frames of silence, 4-byte aligned, in bounded chunks
append the decoded PCM (or one silence frame for DTX)

At close(), tail-pad every track to duration_s so all tracks are identical length. Delete _session_bytes, the reference-uid checks, and the first-packet pad entirely — the per-packet gap logic subsumes all three (a first packet at minute 40 sees a 40-minute gap and pads exactly).

Placement error is bounded by jitter-buffer delivery delay plus the threshold, per talk spurt, and is non-cumulative — every placement is absolute against our own clock. Mutes, leaves, rejoins, reconnects, region changes and packet loss all become silence automatically, because the clock does not care why packets stopped.

The 250 ms threshold is load-bearing: without it, arrival jitter triggers one- and two-frame pads inside continuous speech, which stretches it audibly.

Rejected alternatives

RTP-timestamp anchoring would be frame-accurate and the data is available (.timestamp, .ssrc, .sequence all reach the sink). Deferred, not dismissed: it assumes every Discord client advances its RTP timestamp across silence, and any client that does not would silently recreate this exact bug, invisible until the next bad summary. Ship wall-clock anchoring, log ts_delta_vs_wall_delta in the existing AUDIO_DIAG stream for one release, and revisit only if that data justifies it.

Accepting the jitter-buffer PLC FakePackets (currently skipped at recording.py:163-166) cannot fix this and must not be attempted. PLC fires only on sequence-number gaps, and sequence numbers pause during silence — inter-spurt gaps generate zero FakePackets. It also re-opens a known production incident: commit 02898b3 documents deferred DAVE packets causing a 148x PLC explosion, 340 MB for 12 seconds of audio.

Dependencies

Must ship together with the headerless container change and the SSRC decoder reset in this milestone, and with the VAD change — correcting capture makes every track run the full wall-clock length, which re-arms Whisper's hallucination-on-silence failure and multiplies ASR input roughly ninefold.

Acceptance criteria

PerUserPCMSink takes started_at and an injectable clock
Gap-filling silence is written before each append when the gap exceeds the threshold
All pads are 4-byte aligned (2 channels x s16); a misaligned pad swaps channels and garbles everything after it
Pad targets are clamped to wall-clock elapsed so one corrupt input cannot write gigabytes of silence
close() tail-pads every track to duration_s
_session_bytes, the next(iter(self._writers)) checks and the first-packet pad are deleted
All per-speaker tracks from one recording have the same duration, within 300 ms of the session's wall clock

**Severity: CRITICAL.** Found in the August 2026 session lifecycle review (#319). This is the root cause of the scrambled chronology and misattributed actions reported after the 2026-08-11 session, and it affects **every multi-speaker recording Quest Board has ever produced**. ## The defect Per-speaker WAV tracks are not on a shared timeline. `PerUserPCMSink.write()` appends decoded PCM only when a packet for that user arrives, and Discord clients stop transmitting entirely during silence (roughly five DTX hangover frames, ~100 ms, then nothing). Every pause is therefore deleted from that speaker's track, cumulatively, for hours. A track's duration ends up equal to that speaker's *total talk time*, not the session's wall-clock length. Whisper's timestamps are relative to the file it is given, so each track carries its own private, compressed clock. `merge_attributed_transcript` then sorts segments from six mutually incomparable clocks onto one axis. The one piece of machinery that looks like a session clock is not one. `_session_bytes` is documented at `recording.py:118` as *"Bytes on the first speaker's timeline"* and advances only for packets where `uid == next(iter(self._writers))` — one arbitrary user, whose own track also only advances while they transmit. Late speakers are padded once, on their first packet, to that value. ## Evidence - `bot/questboard_bot/cogs/recording.py:156-215` — `write()` is the only place audio is produced. It never reads `data.packet.timestamp` or `.sequence` and never consults a wall clock. - `bot/questboard_bot/cogs/recording.py:118-119` — the `_session_bytes` comment. - `bot/questboard_bot/cogs/recording.py:172-174` — the one-time first-packet pad. - `bot/questboard_bot/cogs/recording.py:183-184`, `:204-205` — `_session_bytes` advances only for the reference uid. - Commit `7ddff16` ("Fix per-speaker timestamp alignment", Apr 2026) introduced this and its message claims `_session_bytes` "tracks the wall-clock position of the first speaker". It has never read a clock. That commit is the original misdiagnosis. ## Production confirmation Session `56cc4dee-99c0-470a-bd90-18e7621b4194`, ~3 hours scheduled. Stored transcript: 1904 lines, 0 unparsed. | Speaker | Lines | First | Last | |---|---|---|---| | DesertCreosote | 675 | 00:00:04 | **00:54:47** | | Viquilonto (Viq) | 203 | 00:00:03 | 00:15:46 | | Wyatt | 327 | 00:00:01 | 00:14:59 | | Idani | 343 | 00:00:00 | 00:14:45 | | Harrowhark | 223 | 00:00:00 | 00:14:11 | | Clio | 133 | 00:00:00 | 00:10:07 | Max timestamp 00:54:47 for a multi-hour session, per-speaker durations summing to ~2.08 h, and impossible sustained speech density (Idani: one utterance every 2.6 seconds for their entire track). ## Proposed fix — wall-clock anchoring Give `PerUserPCMSink` a `t0` (the same `time.monotonic()` sample as `rec.started_at`, with an injectable `clock` parameter for tests). Per packet, for user `uid`: ``` target_frames = int((clock() - t0) * 48000) gap = target_frames - frames_written[uid] if gap > GAP_THRESHOLD (250 ms worth): write `gap` frames of silence, 4-byte aligned, in bounded chunks append the decoded PCM (or one silence frame for DTX) ``` At `close()`, tail-pad every track to `duration_s` so all tracks are identical length. **Delete `_session_bytes`, the reference-uid checks, and the first-packet pad entirely** — the per-packet gap logic subsumes all three (a first packet at minute 40 sees a 40-minute gap and pads exactly). Placement error is bounded by jitter-buffer delivery delay plus the threshold, per talk spurt, and is **non-cumulative** — every placement is absolute against our own clock. Mutes, leaves, rejoins, reconnects, region changes and packet loss all become silence automatically, because the clock does not care why packets stopped. The 250 ms threshold is load-bearing: without it, arrival jitter triggers one- and two-frame pads *inside* continuous speech, which stretches it audibly. ## Rejected alternatives **RTP-timestamp anchoring** would be frame-accurate and the data is available (`.timestamp`, `.ssrc`, `.sequence` all reach the sink). Deferred, not dismissed: it assumes every Discord client advances its RTP timestamp across silence, and any client that does not would silently recreate this exact bug, invisible until the next bad summary. Ship wall-clock anchoring, log `ts_delta_vs_wall_delta` in the existing `AUDIO_DIAG` stream for one release, and revisit only if that data justifies it. **Accepting the jitter-buffer PLC FakePackets** (currently skipped at `recording.py:163-166`) cannot fix this and must not be attempted. PLC fires only on *sequence-number* gaps, and sequence numbers pause during silence — inter-spurt gaps generate zero FakePackets. It also re-opens a known production incident: commit `02898b3` documents deferred DAVE packets causing a 148x PLC explosion, 340 MB for 12 seconds of audio. ## Dependencies Must ship together with the headerless container change and the SSRC decoder reset in this milestone, and with the VAD change — correcting capture makes every track run the full wall-clock length, which re-arms Whisper's hallucination-on-silence failure and multiplies ASR input roughly ninefold. ## Acceptance criteria - [ ] `PerUserPCMSink` takes `started_at` and an injectable `clock` - [ ] Gap-filling silence is written before each append when the gap exceeds the threshold - [ ] All pads are 4-byte aligned (2 channels x s16); a misaligned pad swaps channels and garbles everything after it - [ ] Pad targets are clamped to wall-clock elapsed so one corrupt input cannot write gigabytes of silence - [ ] `close()` tail-pads every track to `duration_s` - [ ] `_session_bytes`, the `next(iter(self._writers))` checks and the first-packet pad are deleted - [ ] All per-speaker tracks from one recording have the same duration, within 300 ms of the session's wall clock

claude-bot added this to the v3.11.5 - Recording Clock Hotfix milestone

2026-08-25 20:38:40 +00:00

claude-bot added the

bot

recording

bug

labels