perf(bot): stream recording audio to disk instead of buffering in RAM (#82) #167
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/82-stream-audio-to-disk"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
PerUserPCMSinkaccumulated each speaker's silence-padded 48kHz stereo PCM in aBytesIO(≈675 MB/hour/speaker) and only flushed at stop via a full-buffergetvalue()copy — a 5-person/6-hour session ≈ 20 GB resident, OOMing long before themax_recording_hourscap, and a crash lost all audio. This streams each speaker's PCM to an on-disk WAV as it arrives.Changes
cogs/recording.py: the sink opens awavewriter per speaker on their first packet, streams leading silence (in bounded 1s chunks — no multi-GB zero allocation for late joiners),writeframesraws each packet (one 20ms silence frame for DTX), and finalizes headers atclose()(idempotent,_closed-guarded)._processcloses writers defensively and cleans up raw files on error.services/recording_status.py, tests repointed off the removed in-memoryaudio_dataaccessor.convert_to_whisper_wavFFmpeg step (-ar 16000 -ac 1) → same{user_id}.wav, samespeakers.json, same upload.utils/audio.pyuntouched. Timeline/silence arithmetic reproduced exactly.Memory is now O(speakers), independent of duration; no
getvalue()/full-buffer copy remains; partial audio survives a crash.Verification
Full bot suite green on the dev server: 186 passed (+7 new sink write-path tests covering first-packet formatting, late-joiner leading silence, DTX frame, a combined timeline, absence of any in-memory buffer, idempotent close, and empty-session). A live before/after RSS + end-to-end transcription runbook (needs a real Discord session) is documented and will be posted as a PR comment.
Follow-ups noted (out of scope): optional in-flight 16kHz-mono downsample (would let us drop the FFmpeg step) and crash-resume of orphaned raw WAVs.
Closes #82
🤖 Generated with Claude Code
PerUserPCMSink accumulated each speaker's decoded 48 kHz stereo PCM in an io.BytesIO that grew toward full session length and was copied wholesale at stop (getvalue()), doubling peak memory — ~675 MB/hour/speaker, OOMing long before max_recording_hours. A crash also lost all captured audio. The sink now opens a wave.Wave_write per speaker on their first packet and writeframesraw()'s each Opus frame straight to the on-disk raw WAV in the audio_temp volume (leading-silence padding for late joiners is streamed in bounded 1 s chunks; DTX packets still write one 20 ms silence frame; the cross-speaker timeline arithmetic is unchanged). Resident memory is now bounded to the OS file buffer plus one frame regardless of session length, and whatever was flushed survives a crash. Output contract is byte-identical: the streamed file is the same 48 kHz stereo raw WAV (same path {session_id}_{user_id}_raw.wav, same format) the finalize path used to build in RAM, so _process still resamples each to 16 kHz mono via the unchanged FFmpeg step (utils/audio.py) into {session_dir}/{user_id}.wav. The stop-path getvalue()/full-buffer copy is gone; writers are closed in cleanup() (and defensively in _process) with headers finalised on close. audio_data (BytesIO accessor) is replaced by audio_files (user_id -> Path) for the finalize path and bytes_written() (user_id -> bytes) for live status; recording_status.py updated to the new accessor. Tests: added sink write-path coverage (first-packet leading silence, late-joiner gap padding, DTX silence frame, combined timeline, live bytes_written, idempotent close, correctly-formatted on-disk WAV) and repointed the _process/status tests at the on-disk files. 186 bot tests pass. The optional in-flight 48k-stereo->16k-mono downsample and crash-resume are left as follow-ups (out of scope to keep output byte-identical). Closes #82 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>