feat: VAD silence-trim before transcription to cut Whisper time/cost (#101) #168

Merged

claude-bot merged 1 commit from feat/101-vad-silence-trim into main

2026-07-16 00:45:47 +00:00

claude-bot commented

2026-07-16 00:29:19 +00:00

Contributor

Summary

Per-speaker tracks are silence-padded to full session length, so transcription costs ≈ N speakers × session wall-clock even though each person talks a fraction of the time (a 5-person/4-hour session ≈ 20h of mostly-silent audio). This adds optional VAD trimming: detect speech spans, transcribe only those, and offset the results back onto the session timeline.

Design

Segmentation: ffmpeg silencedetect (zero new Python dep) → parse silences → complement to speech spans → pad 300ms/side → merge overlapping. Fully-silent track → 0 Whisper calls.
Timestamp offset (the critical bit): each span is cut to its own 16kHz-mono WAV and transcribed independently (segments come back 0-based), then _offset_segments adds the span's session-relative start to every start/end; merged via the existing merge_attributed_transcript (sorts by start). Proven equivalent to the untrimmed timeline by test.
Whisper call shape: the untrimmed path keeps the server-merged /transcribe/session endpoint; the VAD path issues one /transcribe per span and merges client-side (server-side merge gives no hook for per-span offsets). Same verbose_json segment shape.
Config (Admin → Bot Settings, beside Whisper): vad_trim_enabled (default off), vad_min_silence_ms (2000), vad_noise_floor_db (-30). Surfaced through the bot-settings schema/GET/PUT and a frontend toggle + inputs.
OFF is byte-identical: a thin selector calls the unchanged transcribe_session when disabled/None — no ffmpeg, segments pass through by identity. Tested.
Logs "VAD trimmed 14.2h -> 3.1h of audio" when the VAD path runs.

Note: Dockerfile change (out of original scope, but required)

ffmpeg was only in bot-base, but the Celery audio worker runs the backend image — so silencedetect would fail at runtime. Added ffmpeg to backend-base. Slightly widens the backend image; the CI docker job validates the build.

Verification

Backend full suite 382 passed (+15 VAD tests incl. the timeline-equivalence and OFF-path-unchanged proofs); frontend Admin tests 9/9, full frontend 67/67 (one pre-existing unrelated react-markdown import failure in an untouched suite). ruff 0.4.4 clean.

Closes #101

🤖 Generated with Claude Code

## Summary Per-speaker tracks are silence-padded to full session length, so transcription costs ≈ N speakers × session wall-clock even though each person talks a fraction of the time (a 5-person/4-hour session ≈ 20h of mostly-silent audio). This adds optional VAD trimming: detect speech spans, transcribe only those, and offset the results back onto the session timeline. ## Design - **Segmentation**: `ffmpeg silencedetect` (zero new Python dep) → parse silences → complement to speech spans → pad 300ms/side → merge overlapping. Fully-silent track → 0 Whisper calls. - **Timestamp offset (the critical bit)**: each span is cut to its own 16kHz-mono WAV and transcribed independently (segments come back 0-based), then `_offset_segments` adds the span's session-relative start to every `start`/`end`; merged via the existing `merge_attributed_transcript` (sorts by start). Proven equivalent to the untrimmed timeline by test. - **Whisper call shape**: the untrimmed path keeps the server-merged `/transcribe/session` endpoint; the VAD path issues one `/transcribe` per span and merges client-side (server-side merge gives no hook for per-span offsets). Same `verbose_json` segment shape. - **Config** (Admin → Bot Settings, beside Whisper): `vad_trim_enabled` (**default off**), `vad_min_silence_ms` (2000), `vad_noise_floor_db` (-30). Surfaced through the bot-settings schema/GET/PUT and a frontend toggle + inputs. - **OFF is byte-identical**: a thin selector calls the unchanged `transcribe_session` when disabled/None — no ffmpeg, segments pass through by identity. Tested. - Logs `"VAD trimmed 14.2h -> 3.1h of audio"` when the VAD path runs. ## Note: Dockerfile change (out of original scope, but required) `ffmpeg` was only in `bot-base`, but the Celery **audio worker runs the backend image** — so `silencedetect` would fail at runtime. Added `ffmpeg` to `backend-base`. Slightly widens the backend image; the CI `docker` job validates the build. ## Verification Backend full suite **382 passed** (+15 VAD tests incl. the timeline-equivalence and OFF-path-unchanged proofs); frontend Admin tests 9/9, full frontend 67/67 (one pre-existing unrelated `react-markdown` import failure in an untouched suite). ruff 0.4.4 clean. Closes #101 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit

2026-07-16 00:29:20 +00:00

feat(audio): trim silence before transcription (VAD) to cut Whisper cost

CI / Frontend tests, audit, and build (pull_request) Successful in 1m26s

Details

CI / Backend lint (ruff) (pull_request) Successful in 48s

Details

CI / Bot tests and audit (pull_request) Successful in 2m27s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 4m8s

Details

CI / Docker image build (pull_request) Successful in 13m9s

Details

03a17fc181

The bot uploads per-speaker WAV tracks silence-padded to the full session
length, so the transcribe path sent each full-length track to Whisper —
cost scaled with N speakers x session wall-clock even though each speaker
talks a fraction of the time.

When the new (default-off) VAD toggle is enabled, each track is scanned
with ffmpeg silencedetect, cut down to just its speech spans (with 300ms
padding and sub-threshold gaps folded in), and transcribed per-span via the
single-file /transcribe endpoint. Every returned segment is offset by its
span's session-relative start time before merging, so the merged transcript
(ordering, timings, speaker attribution) is indistinguishable from the
untrimmed path. Per-session audio reduction is logged.

The Whisper call shape changes only for the VAD path: it uses per-segment
/transcribe requests merged client-side (the multi-track /transcribe/session
endpoint merges server-side and can't apply per-span offsets). With the
toggle off, the code calls transcribe_session exactly as before —
byte-identical, proven by a test.

Config lives alongside the existing Whisper settings in Admin -> Bot
Settings: vad_trim_enabled (default off), vad_min_silence_ms (default 2000),
vad_noise_floor_db (default -30). ffmpeg is added to the backend image (it
was only in the bot image) since the audio worker now runs it.

Closes #101

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed

2026-07-16 00:29:43 +00:00

claude-bot merged commit 5681d5a998 into main

2026-07-16 00:45:47 +00:00

claude-bot deleted branch feat/101-vad-silence-trim

2026-07-16 00:45:48 +00:00

claude-bot referenced this pull request from a commit

2026-07-16 00:45:49 +00:00

Merge pull request 'feat: VAD silence-trim before transcription to cut Whisper time/cost (#101)' (#168) from feat/101-vad-silence-trim into main