rbrooks/Quest-Board

Watch

Fork 0

Transcription: send campaign names as hotwords by default, and check it on dev with a real recording #1315

New issue

Closed

opened 2026-11-02 17:26:43 +00:00 by claude-bot · 4 comments

claude-bot commented

2026-11-02 17:26:43 +00:00

Collaborator

This is the owner's answer to Q203 (TR1, 2 Nov), which followed #506's measurement in PR #1313. With a realistic 100-name list, hotwords got 89–90% of names right, against 80–82% for initial_prompt. At 24 names the two were the same, and sending both made WhisperX return 500.

The owner chose option (a): switch the default to hotwords when the server supports it, fall back to the prompt, and check it on dev with a real campaign's recording before 4.7.0 ships.

The change:

QB_ASR_VOCABULARY_STYLE now defaults to hotwords.
A server that doesn't declare hotwords gets initial_prompt. That fallback is quiet (logged at info) when the value is the default, and keeps its warning when an operator set hotwords explicitly.
initial_prompt stays available to keep the old behaviour.
Tests, .env.example, OPERATIONS.md and a changelog fragment are updated.

The dev check, before 4.7.0 ships. On dev, transcribe one real session's kept audio with each style and compare how many campaign names come back right. It must never overwrite that session's transcript: run the ASR directly from the worker container against a copy of the audio and score it, rather than reprocessing the session. Record the result here.

This is the owner's answer to Q203 (TR1, 2 Nov), which followed #506's measurement in PR #1313. With a realistic 100-name list, `hotwords` got 89–90% of names right, against 80–82% for `initial_prompt`. At 24 names the two were the same, and sending both made WhisperX return 500. The owner chose option (a): switch the default to hotwords when the server supports it, fall back to the prompt, and **check it on dev with a real campaign's recording before 4.7.0 ships**. The change: - `QB_ASR_VOCABULARY_STYLE` now defaults to `hotwords`. - A server that doesn't declare hotwords gets `initial_prompt`. That fallback is quiet (logged at info) when the value is the default, and keeps its warning when an operator set `hotwords` explicitly. - `initial_prompt` stays available to keep the old behaviour. - Tests, `.env.example`, OPERATIONS.md and a changelog fragment are updated. **The dev check, before 4.7.0 ships.** On dev, transcribe one real session's kept audio with each style and compare how many campaign names come back right. It must never overwrite that session's transcript: run the ASR directly from the worker container against a copy of the audio and score it, rather than reprocessing the session. Record the result here.

claude-bot referenced this issue

2026-11-02 17:31:38 +00:00

feat(webapp): campaign names go to the transcriber as hotwords by default (#1315) #1316

rbrooks referenced this issue from a commit

2026-11-02 17:31:40 +00:00

feat(webapp): campaign names go to the transcriber as hotwords by default (#1315)

rbrooks referenced this issue from a commit

2026-11-02 17:31:40 +00:00

test: the vocabulary tests expect hotwords from a server that offers them (#1315)

claude-bot referenced this issue from a commit

2026-11-02 18:49:18 +00:00

Merge pull request 'feat(webapp): campaign names go to the transcriber as hotwords by default (#1315)' (#1316) from feat/1315-hotwords-default into main

claude-bot commented

2026-11-02 18:49:48 +00:00

Author

Collaborator

The code is merged in #1316, on the owner's say (2 Nov). hotwords is now the default, falling back quietly to initial_prompt on a server that doesn't declare hotwords.

Still open: the dev check with a real recording, before 4.7.0 ships. This issue stays open until it's recorded here.

The code is merged in #1316, on the owner's say (2 Nov). `hotwords` is now the default, falling back quietly to `initial_prompt` on a server that doesn't declare hotwords. Still open: **the dev check with a real recording, before 4.7.0 ships.** This issue stays open until it's recorded here.

claude-bot commented

2026-11-02 18:52:54 +00:00

Author

Collaborator

The dev check can't run on dev. Dev's bot has no Discord token, so dev has never recorded a session. Its only kept audio is the synthetic ic_only scene from 5 Sep, which #1313 already measured. WhisperX's /health does declare both mechanisms.

The only real kept audio is on prod, and reading it needs the owner's say. I asked on the Questions page as Q204 (TR2). If the answer is yes, the method stays as planned: copy the audio off read-only, transcribe it both ways from a throwaway container, report counts only, and never touch the session.

The dev check can't run on dev. Dev's bot has no Discord token, so dev has never recorded a session. Its only kept audio is the synthetic `ic_only` scene from 5 Sep, which #1313 already measured. WhisperX's `/health` does declare both mechanisms. The only real kept audio is on prod, and reading it needs the owner's say. I asked on the Questions page as Q204 (TR2). If the answer is yes, the method stays as planned: copy the audio off read-only, transcribe it both ways from a throwaway container, report counts only, and never touch the session.

claude-bot commented

2026-11-02 18:58:36 +00:00

Author

Collaborator

Real-recording check on prod audio (owner-approved, 2 Nov)

Read-only throughout. I copied three 10-minute clips off prod and transcribed each one twice against WhisperX on the GPU box, once per arm. I used the same form fields as the product: speaker, language=en, and the vocabulary in one field. The copies and the transcripts have been deleted. The session, its audio and its transcript were never touched.

Setup

The most recent kept session, 5.7 days old: 5 speakers, 83 minutes each.
The 3 busiest speaker windows, 10 minutes each, 30 minutes of audio in all.
The campaign's vocabulary, as the product builds it (campaign_vocabulary, limit 100): 43 terms.

Arm	Terms heard (of 43)	Mentions	Near-misses*	Words	HTTP errors
`initial_prompt`	3	6	131	1678	0
`hotwords`	4	7	132	1677	0

* The near-miss count is noisy. It counts any 4+ letter word within edit distance 2 of a one-word term, and short names collide with common words, so it can't separate the two arms.

Verdict: no regression on real audio. hotwords heard one more term and one more mention, the word counts are effectively identical, and there were no errors. The difference is too small to call a gain, because names were rarely spoken in these 30 minutes. The synthetic measurement (#1313) remains the evidence for the gain on a long list. As a default, hotwords is safe to ship.

## Real-recording check on prod audio (owner-approved, 2 Nov) Read-only throughout. I copied three 10-minute clips off prod and transcribed each one twice against WhisperX on the GPU box, once per arm. I used the same form fields as the product: `speaker`, `language=en`, and the vocabulary in one field. The copies and the transcripts have been deleted. The session, its audio and its transcript were never touched. **Setup** - The most recent kept session, 5.7 days old: 5 speakers, 83 minutes each. - The 3 busiest speaker windows, 10 minutes each, 30 minutes of audio in all. - The campaign's vocabulary, as the product builds it (`campaign_vocabulary`, limit 100): **43 terms**. | Arm | Terms heard (of 43) | Mentions | Near-misses* | Words | HTTP errors | |---|---|---|---|---|---| | `initial_prompt` | 3 | 6 | 131 | 1678 | 0 | | `hotwords` | 4 | 7 | 132 | 1677 | 0 | \* The near-miss count is noisy. It counts any 4+ letter word within edit distance 2 of a one-word term, and short names collide with common words, so it can't separate the two arms. **Verdict:** no regression on real audio. `hotwords` heard one more term and one more mention, the word counts are effectively identical, and there were no errors. The difference is too small to call a gain, because names were rarely spoken in these 30 minutes. The synthetic measurement (#1313) remains the evidence for the gain on a long list. As a default, `hotwords` is safe to ship.

claude-bot commented

2026-11-02 18:59:37 +00:00

Author

Collaborator

The real-recording check is done, on prod's kept audio with the owner's approval. Counts are in the comment above.

Hotwords did no worse than the prompt: 4 of 43 names heard against 3, the same word count, and no errors. But those 30 minutes held too few spoken names to show the gain, so the evidence for a gain is still the synthetic matrix (#1313).

That meets the owner's condition, "check it on dev with a real recording before 4.7.0 ships": the default is safe on real audio. Closing.

If a stronger real-audio comparison is wanted later, the harness's summary-leg --transcript can score a name-heavy session once one is kept.

The real-recording check is done, on prod's kept audio with the owner's approval. Counts are in the comment above. Hotwords did no worse than the prompt: 4 of 43 names heard against 3, the same word count, and no errors. But those 30 minutes held too few spoken names to show the gain, so the evidence for a gain is still the synthetic matrix (#1313). That meets the owner's condition, "check it on dev with a real recording before 4.7.0 ships": the default is safe on real audio. Closing. If a stronger real-audio comparison is wanted later, the harness's `summary-leg --transcript` can score a name-heavy session once one is kept.

claude-bot closed this issue

2026-11-02 18:59:39 +00:00