[Ops] Trim silence before transcription (VAD) to cut Whisper time and cost #101

New issue

Closed

opened 2026-07-14 19:49:08 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-07-14 19:49:08 +00:00

Contributor

Context

The bot uploads per-speaker WAV tracks that are silence-padded to full session length so all tracks share one timeline. The backend's audio processing task (webapp/backend/app/tasks/reminder_tasks.py, transcribe path around :1475-1610, Whisper endpoint config at :1592-1608) sends each full-length track to Whisper.

Motivation

Transcribing N speakers costs ≈ N × the session's wall-clock minutes of audio, even though each speaker talks only a fraction of the time. In a 5-person 4-hour session that's ~20 hours of audio, most of it silence — the dominant compute cost for self-hosted Whisper on the GPU server and the dominant dollar cost for any hosted transcription.

Fix / Spec

In the transcribe path, before sending each per-speaker track to Whisper:
- Detect speech spans (ffmpeg silencedetect on the WAV is the zero-new-dependency option; silero-vad or webrtcvad acceptable alternatives).
- Cut the track into speech-only segments (retain a small padding margin, e.g. 200-500ms each side, and merge spans closer than the minimum-silence threshold).
- Transcribe the segments.
Critical correctness requirement: OFFSET every returned Whisper segment's timestamps by its source span's start time, so the merged transcript timeline remains session-relative and interleaves correctly across speakers. The merged output must be indistinguishable in ordering/timing from the untrimmed path.
Config: an admin toggle (default on or off — decide, default-off is safer for rollout) plus a minimum-silence-duration threshold with sane defaults, in Admin → Bot Settings alongside the existing Whisper config.
Verify the current Whisper endpoint contract: if the existing single-request per-track call can't take multiple segments, switch to per-segment requests and merge results (respecting the verbose_json segment format already consumed at :1475).
Log the measured reduction per session, e.g. "VAD trimmed 14.2h → 3.1h of audio".

Acceptance criteria

A test (or recorded fixture) proves merged transcript ordering and timestamps are unchanged versus the untrimmed path.
Toggle off → behavior is byte-identical to today.
Trimmed-minutes reduction is logged per session.
No regression in speaker attribution in the merged transcript.

References

webapp/backend/app/tasks/reminder_tasks.py:1475-1610 (transcribe path), :1592-1608 (Whisper config/endpoint call)

Filed from the July 2026 full-project review.

## Context The bot uploads per-speaker WAV tracks that are silence-padded to full session length so all tracks share one timeline. The backend's audio processing task (`webapp/backend/app/tasks/reminder_tasks.py`, transcribe path around `:1475-1610`, Whisper endpoint config at `:1592-1608`) sends each full-length track to Whisper. ## Motivation Transcribing N speakers costs ≈ N × the session's wall-clock minutes of audio, even though each speaker talks only a fraction of the time. In a 5-person 4-hour session that's ~20 hours of audio, most of it silence — the dominant compute cost for self-hosted Whisper on the GPU server and the dominant dollar cost for any hosted transcription. ## Fix / Spec 1. In the transcribe path, before sending each per-speaker track to Whisper: - Detect speech spans (ffmpeg `silencedetect` on the WAV is the zero-new-dependency option; `silero-vad` or `webrtcvad` acceptable alternatives). - Cut the track into speech-only segments (retain a small padding margin, e.g. 200-500ms each side, and merge spans closer than the minimum-silence threshold). - Transcribe the segments. 2. **Critical correctness requirement**: OFFSET every returned Whisper segment's timestamps by its source span's start time, so the merged transcript timeline remains session-relative and interleaves correctly across speakers. The merged output must be indistinguishable in ordering/timing from the untrimmed path. 3. Config: an admin toggle (default on or off — decide, default-off is safer for rollout) plus a minimum-silence-duration threshold with sane defaults, in Admin → Bot Settings alongside the existing Whisper config. 4. Verify the current Whisper endpoint contract: if the existing single-request per-track call can't take multiple segments, switch to per-segment requests and merge results (respecting the verbose_json segment format already consumed at `:1475`). 5. Log the measured reduction per session, e.g. `"VAD trimmed 14.2h → 3.1h of audio"`. ## Acceptance criteria - [ ] A test (or recorded fixture) proves merged transcript ordering and timestamps are unchanged versus the untrimmed path. - [ ] Toggle off → behavior is byte-identical to today. - [ ] Trimmed-minutes reduction is logged per session. - [ ] No regression in speaker attribution in the merged transcript. ## References - `webapp/backend/app/tasks/reminder_tasks.py:1475-1610` (transcribe path), `:1592-1608` (Whisper config/endpoint call) _Filed from the July 2026 full-project review._

claude-bot added the

backend

enhancement

recording

labels

2026-07-14 19:49:16 +00:00

claude-bot added this to the v3.4.0 - Reliability, Operations & Self-Hosting milestone

2026-07-14 19:49:17 +00:00