Spike: live/streaming transcription — options & viability (accuracy vs speed) #238

New issue

Open

opened 2026-07-21 21:03:47 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-07-21 21:03:47 +00:00

Contributor

Purpose

De-risk the Shelf Listens layer by evaluating several concrete approaches to live / near-real-time transcription on the actual self-hosted GPU, scored on accuracy vs. speed (they trade off directly). ⚠️ This deserves a dedicated session — do not start until v3.12 is picked up.

Current state: the bot records per-speaker buffers and transcribes only on stop, via a remote WhisperX endpoint. The live layer needs chunked/streaming transcription during play.

Options to evaluate

Chunked WhisperX — flush each speaker's rolling buffer every ~15–30s (or on a VAD silence boundary) and batch-transcribe the chunk. Reuses the existing stack; latency ≈ chunk size.
faster-whisper chunked server — WhisperX's CTranslate2 backend, but skip alignment/diarization (we already have per-speaker audio). Lighter/simpler than full WhisperX for the live pass.
Purpose-built streaming ASR — whisper_streaming (UFAL LocalAgreement), WhisperLive, or a rolling-window faster-whisper server, for lower latency than fixed chunks.
Dual-model split — small/fast model (distil-whisper / medium) for the live entity pass + large-v3 (WhisperX) for the authoritative post-session transcript. Likely the pragmatic winner.
Wake-word + tiny ASR — only if we pursue hands-free voice commands (a much harder latency target than entity auto-surface; probably a separate deliverable).

Evaluation axes / method

Latency — real end-to-end (spoken → card surfaced) per option, measured on the dev GPU with a representative TTRPG voice clip.
Accuracy — WER, but more importantly entity precision/recall: do proper nouns / NPC names come out matchable against the wiki index? Names matter more than perfect prose.
GPU load / concurrency — continuous live load, and contention if a post-session batch + Ollama run simultaneously.
Self-host complexity — what a self-hoster must stand up and configure.

Deliverable

A short recommendation (likely option 4) backed by real numbers, so v3.12 commits to a proven approach rather than an assumption. Note the different bars: entity auto-surface tolerates ~15–30s latency; hands-free voice does not.

🤖 Generated with Claude Code

## Purpose De-risk the *Shelf Listens* layer by evaluating **several concrete approaches to live / near-real-time transcription** on the **actual self-hosted GPU**, scored on **accuracy vs. speed** (they trade off directly). ⚠️ **This deserves a dedicated session — do not start until v3.12 is picked up.** **Current state:** the bot records per-speaker buffers and transcribes **only on stop**, via a remote WhisperX endpoint. The live layer needs chunked/streaming transcription *during* play. ## Options to evaluate 1. **Chunked WhisperX** — flush each speaker's rolling buffer every ~15–30s (or on a VAD silence boundary) and batch-transcribe the chunk. Reuses the existing stack; latency ≈ chunk size. 2. **faster-whisper chunked server** — WhisperX's CTranslate2 backend, but **skip alignment/diarization** (we already have per-speaker audio). Lighter/simpler than full WhisperX for the live pass. 3. **Purpose-built streaming ASR** — `whisper_streaming` (UFAL LocalAgreement), WhisperLive, or a rolling-window faster-whisper server, for lower latency than fixed chunks. 4. **Dual-model split** — small/fast model (distil-whisper / medium) for the live entity pass **+** large-v3 (WhisperX) for the authoritative post-session transcript. Likely the pragmatic winner. 5. **Wake-word + tiny ASR** — only if we pursue hands-free voice commands (a much harder latency target than entity auto-surface; probably a separate deliverable). ## Evaluation axes / method - **Latency** — real end-to-end (spoken → card surfaced) per option, measured on the dev GPU with a representative TTRPG voice clip. - **Accuracy** — WER, but more importantly **entity precision/recall**: do proper nouns / NPC names come out matchable against the wiki index? Names matter more than perfect prose. - **GPU load / concurrency** — continuous live load, and contention if a post-session batch + Ollama run simultaneously. - **Self-host complexity** — what a self-hoster must stand up and configure. ## Deliverable A short recommendation (likely option 4) backed by **real numbers**, so v3.12 commits to a proven approach rather than an assumption. Note the different bars: entity auto-surface tolerates ~15–30s latency; hands-free voice does not. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added this to the (deleted) milestone

2026-07-21 21:03:58 +00:00

claude-bot referenced this issue

2026-07-27 22:52:44 +00:00

Epic: The Shelf Listens — transcript-driven auto-surface #237

claude-bot modified the milestone from (deleted) to Backlog - Speculative / Long-Term

2026-07-27 22:52:54 +00:00