v3.11.5 — per-speaker recordings never shared a session clock (#320) #418
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/recording-wall-clock"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #320, #321, #322, #323, #324, #325, #326, #327, #328, #417. Tracking issue: #319.
The bug
Per-speaker audio tracks have never been on a shared timeline.
PerUserPCMSinkappended decoded PCM as packets arrived, and Discord clients stop transmitting entirely during silence — so every pause was deleted from every track. A track ended up as long as its speaker's total talk time rather than the session, and because Whisper timestamps are relative to the file it is given, each speaker carried a private, compressed clock.merge_attributed_transcriptthen sorted six mutually incomparable clocks onto one axis.The thing that looked like a session clock was not one:
_session_bytesis documented as "bytes on the first speaker's timeline" and advanced only for one arbitrary user, whose own track also only advanced while they transmitted. Commit7ddff16claimed it tracked wall-clock position; it never read a clock.This affected every multi-speaker recording ever produced, not one bad session.
Production evidence
Session
56cc4dee— ~3 hours scheduled. Stored transcript, 1904 lines, 0 unparsed:Max timestamp 00:54:47 for a multi-hour session; per-speaker spans summing to ~2.08 h; and impossible sustained speech density (Idani at one utterance every 2.6 seconds for their whole track).
Note what is not evidence: the transcript's timestamps are perfectly monotonic. The merge sorts by start time, so output is non-decreasing for any input — that validates the
sorted()call, not the data. The detectable signal was always magnitude.Before/after, measured
Driving the pre-fix sink through a simulated 30-minute session with realistic talk shares, then the same simulation through the fixed sink:
The pre-fix reference speaker lands at exactly their talk time — the same signature as DesertCreosote's 00:54:47 in the real transcript.
The fix
Placement is now absolute rather than incremental. Before appending a packet the sink computes where it belongs from
clock() - t0and pads that speaker's file with silence to close the gap._session_bytes, the reference-user checks and the first-packet pad are deleted — the per-packet gap logic subsumes all three. Error per talk spurt is bounded by jitter-buffer delay plus a 250 ms threshold and is non-cumulative, so it cannot grow over a four-hour session. Mutes, leaves, rejoins, reconnects and packet loss all become silence for free, because the clock does not care why packets stopped.RTP-timestamp anchoring would be frame-accurate and the data is available, but it assumes every client advances its timestamp across silence — any client that does not would silently recreate this bug. Deferred behind one release of
AUDIO_DIAGlogging.Accepting the jitter-buffer PLC FakePackets cannot fix it and must not be attempted: PLC fires only on sequence-number gaps, which do not occur across silence, and
02898b3documents the 148× PLC explosion that is why the skip exists.Required companions, not nice-to-haves
.s16lecapture (#321). Gap-filled 48 kHz stereo is 192 KB/s of wall clock, reaching 4.15 GB at the six-hour cap and overflowing WAV's 4 GB RIFF size field — failing atclose(), after capturing the whole session.Why nothing caught it
process_audioalready received the wall-clockduration_seconds, logged it into the Discord embed, and compared it against nothing. Two guards now fail loud into the existing failed-status path (#324) — one before transcription so a bad recording costs no GPU time, one before the merge. Both skip when the duration is unknown, since reprocess endpoints pass 0.The live dashboard has been publishing captured time next to elapsed time since the feature shipped, with nothing comparing them. It now warns (#327), and
close()logs a per-speaker captured/expected ratio — one grep after the first real session proves the fix, which matters because the dev stack has no Discord token and cannot exercise the bot end to end.Also in here
#417 — seven fields the session API declared but never sent. Found while adding
pre_timeline_fixto the same builder. Among themlore_generation_status, which the session page gates its whole lore block on, so the progress display, the failure/re-run message, and the only link to the wiki proposals queue could never render. Pulled into the hotfix rather than deferred: a fix that adds one field to a builder while leaving six broken beside it — without the guard that would have caught its own near-miss — is not a fix. The guard is source-level and deliberately not a value comparison, because a value test only catches fields a test author remembered to set. It failed on first run and found two fields (series_id,series_occurrence_date) that reading the code by hand had missed.Two things caught while in there:
erasure_noteswere shipping the erased member'smember_user_idalongside a deliberately anonymous notice shown to every member, and the "Review proposals" link pointed at the wiki home (taken from #375; the rest of its scope is untouched).#328 — the nine already-affected sessions are marked. They cannot be repaired: the audio is reaped and reprocessing would re-derive the same scrambled merge. Backfilled by migration for every session that already had a transcript, which needs no date comparison because the fix ships in the same release. Surfaced as an explanation rather than a bare badge, and shown to players too. Content is not hidden — the events happened, and at least one summary has been hand-corrected.
#325 — transcript text out of the logs. The transcription response was logged at INFO including its opening segments with speaker names, against the invariant in that module's own docstring. Verified present in production logs.
#326 — the tests that asserted the broken model are rewritten. The sink takes an injectable clock;
test_track_duration_equals_wall_clockis the core invariant and fails against the pre-fix sink three independent ways. Companions cover reference-independence, leave/rejoin, SSRC reset, sub-threshold jitter not stretching speech, frame alignment, and thecleanup()fallback. All pure-unit, no Discord.Upgrade note
BOT_CONTRACT_VERSIONstays 1 and nothing refuses to start, but backend and bot must be upgraded together. New backend + old bot now rejects recordings (correctly). New bot + old backend fails quietly instead: full-length tracks with trimming still off. One migration,f1a2b3c4d5e7.Verification
Bot 199 passed · backend 914 passed · frontend 431 passed + eslint clean · ruff check and format clean ·
check_version_syncOK at 3.11.5 · migration applies to a single head.A repo-wide audit for other instances of the #417 class found none — one apparent hit was a false positive from a locally-defined
SessionListIteminrouters/bot.py. The mirror check (frontend reading session fields the API never sends) is also clean.Before deploying to production
🤖 Generated with Claude Code