rbrooks/Quest-Board
Watch
1
0
Fork
You've already forked Quest-Board
 0
Code Issues 58 Pull requests 7 Projects Releases 76 Packages Wiki Activity Actions

feat(repo): eval harness leg 3 in one command, and the hotwords matrix (#506) #1313

Merged
claude-bot merged 4 commits from feat/506-leg3-hotwords into main 2026-11-02 18:49:23 +00:00
Conversation 0 Commits 4 Files changed 6 +864 -43
claude-bot commented 2026-11-02 08:03:00 +00:00
Collaborator
Copy link

Closes #506 (items 3 and 4; items 1, 2 and 5 shipped in #510). This is harness and measurement work only: no product code changes, and the bot contract is untouched.

Item 4: leg 3 in one command.

  • python -m evals.synthetic_session summary-leg [--provider stub|live] [--baseline] [--transcript F] [--vocabulary none|markers|file:P] [--vocabulary-mechanism initial_prompt|hotwords|both] [--record] [--out D].
  • It runs audio → ASR → summary → scores, reusing leg 2's own runner and scorers.
  • It writes <stem>.summary-leg.json with a baseline and delta against leg 2.
  • --transcript skips the ASR step.
  • 14 offline tests, using the stub provider and a mock transport.

Item 3: hotwords against initial_prompt. This ran against the real WhisperX on the GPU box, one run per cell, from docker-test.

Vocabulary Mechanism full markers full WER ic_only markers ic_only WER
none n/a 72.2% 3.73% 70.0% 3.32%
markers (24) initial_prompt 96.3% 3.70% 96.0% 2.56%
markers (24) hotwords 96.3% 3.70% 96.0% 2.56%
markers (24) both 96.3% 3.70% 96.0% 2.56%
diluted (100) initial_prompt 81.5% 3.39% 80.0% 2.78%
diluted (100) hotwords 88.9% 3.13% 90.0% 2.56%
diluted (100) both HTTP 500 HTTP 500
  • With a realistic 100-term list, hotwords adds +7.4 and +10 points of name recall, with lower WER. At 24 terms the two mechanisms are the same.
  • Sending both fields with 100 terms makes WhisperX return 500 every time; this was reproduced with curl.
  • Recommendation: prefer hotwords when the server declares it, fall back to initial_prompt, and never send both. The adapter's default (QB_ASR_VOCABULARY_STYLE=initial_prompt) is not changed here. That is a product decision, and it's for the owner.
  • The leg-3 summary scores are single samples on a six-event fixture, so they're too noisy to rank the mechanisms. --repeat is the follow-up.

Verified on docker-test:

  • lint is clean (985 files formatted, all checks pass);
  • the synthetic suite's four modules plus the new file: 121 passed;
  • the version-sync and changelog checks pass.

🤖 Generated with Claude Code

Closes #506 (items 3 and 4; items 1, 2 and 5 shipped in #510). This is harness and measurement work only: no product code changes, and the bot contract is untouched. **Item 4: leg 3 in one command.** - `python -m evals.synthetic_session summary-leg [--provider stub|live] [--baseline] [--transcript F] [--vocabulary none|markers|file:P] [--vocabulary-mechanism initial_prompt|hotwords|both] [--record] [--out D]`. - It runs audio → ASR → summary → scores, reusing leg 2's own runner and scorers. - It writes `<stem>.summary-leg.json` with a baseline and delta against leg 2. - `--transcript` skips the ASR step. - 14 offline tests, using the stub provider and a mock transport. **Item 3: `hotwords` against `initial_prompt`.** This ran against the real WhisperX on the GPU box, one run per cell, from docker-test. | Vocabulary | Mechanism | `full` markers | `full` WER | `ic_only` markers | `ic_only` WER | |---|---|---|---|---|---| | none | n/a | 72.2% | 3.73% | 70.0% | 3.32% | | markers (24) | initial_prompt | 96.3% | 3.70% | 96.0% | 2.56% | | markers (24) | hotwords | 96.3% | 3.70% | 96.0% | 2.56% | | markers (24) | both | 96.3% | 3.70% | 96.0% | 2.56% | | diluted (100) | initial_prompt | 81.5% | 3.39% | 80.0% | 2.78% | | diluted (100) | hotwords | **88.9%** | **3.13%** | **90.0%** | 2.56% | | diluted (100) | both | HTTP 500 | | HTTP 500 | | - With a realistic 100-term list, `hotwords` adds +7.4 and +10 points of name recall, with lower WER. At 24 terms the two mechanisms are the same. - Sending both fields with 100 terms makes WhisperX return 500 every time; this was reproduced with curl. - **Recommendation:** prefer `hotwords` when the server declares it, fall back to `initial_prompt`, and never send both. The adapter's default (`QB_ASR_VOCABULARY_STYLE=initial_prompt`) is **not** changed here. That is a product decision, and it's for the owner. - The leg-3 summary scores are single samples on a six-event fixture, so they're too noisy to rank the mechanisms. `--repeat` is the follow-up. **Verified on docker-test:** - lint is clean (985 files formatted, all checks pass); - the synthetic suite's four modules plus the new file: 121 passed; - the version-sync and changelog checks pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
summary-leg runs audio -> ASR -> transcript -> LLM summary -> scores through
leg 2's own run_fixture. asr-leg and summary-leg take
--vocabulary-mechanism {initial_prompt,hotwords,both}. README records the
measured matrix and recommendation.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
style: ruff format the leg 3 tests (#506)
All checks were successful
CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 1m1s
Details
CI / Backend lint (ruff) (pull_request) Successful in 37s
Details
CI / Bot/backend version sync (pull_request) Successful in 31s
Details
CI / Docker image build (pull_request) Successful in 21s
Details
CI / Feedback site tests (pull_request) Successful in 1m4s
Details
CI / Does this change touch the recording path? (pull_request) Successful in 21s
Details
CI / Which journeys does this change need? (pull_request) Successful in 19s
Details
CI / Bot tests and audit (pull_request) Successful in 2m20s
Details
CI / Synthetic session harness (no GPU, no LLM) (pull_request) Successful in 8m38s
Details
CI / End-to-end journeys (desktop) (pull_request) Successful in 11m7s
Details
CI / Frontend (shard 2/2) (pull_request) Successful in 17m22s
Details
CI / Frontend (shard 1/2) (pull_request) Successful in 21m14s
Details
CI / Backend migration, tests, and audit (pull_request) Successful in 53m40s
Details
6ba9a67b9f 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Issue related to the backend

  
bot
Issue related to the bot

  
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
frontend
Issue related to the frontend

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
recording
Issue related to the recording system

  
security
Security hardening / vulnerability work

  
wontfix
This won't be fixed

No labels
backend
bot
bug
duplicate
enhancement
frontend
help wanted
invalid
question
recording
security
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/Quest-Board!1313
No description provided.