rbrooks/Quest-Board
Watch
1
0
Fork
You've already forked Quest-Board
 0
Code Issues 58 Pull requests 7 Projects Releases 76 Packages Wiki Activity Actions

test(repo): halve the backend suite, block unmocked network in tests (#937) #938

Merged
claude-bot merged 3 commits from test/937-backend-suite-speed into main 2026-09-29 17:48:26 +00:00
Conversation 0 Commits 3 Files changed 18 +152 -5
claude-bot commented 2026-09-29 17:16:25 +00:00
Collaborator
Copy link

Closes #937.

The default backend run goes from 1,061 s to 536 s locally, with the same tests passing.

1. Stub the local-Whisper /health probe by default.
LocalWhisperProvider.submit() probes /health before transcribing. Tests mocked the transcribe call but not the probe, so about 30 recording-pipeline tests each paid a failing DNS lookup (~3 s) plus the 5 s _HEALTH_TIMEOUT, then silently fell back to v1. The fixture that clears the capability cache (#352) meant every test paid it again. An autouse fixture now returns the probe's own fallback value directly. Tests that exercise the probe carry @pytest.mark.real_asr_probe and mock the HTTP themselves: two whole modules, plus individual tests in six others.

2. A network guard.
An autouse fixture wraps socket.getaddrinfo. Resolving anything other than loopback, an IP literal or the test database's host raises gaierror immediately, and the test fails at teardown naming the host, even when the code under test swallowed the error, as the probe did. The opt-out is @pytest.mark.allow_network; no test needed it. It caught one real case: test_a_hostname_that_does_not_resolve_says_so depended on live DNS and now mocks the failure. There's no new dependency, because pytest-socket guards connect(), and the lookup is where most of the time went.

3. Synthetic-session harness out of the default run.
The four test_synthetic_session_* modules are marked synthetic, and the backend CI job runs pytest -x -q -m "not synthetic". The synthetic-session job is unchanged: it passes explicit files and there's no addopts, so it still runs all 105 tests. They stay serial, per #684.

Also updated: the /qb-test recipe (backend deselects synthetic, plus a new synthetic section) and a changelog fragment.

Verified locally:

  • Default run: 5769 passed, 14 skipped, 105 deselected in 536 s.
  • -m synthetic: 105 passed in 294 s.
  • ruff 0.4.4 format and check: clean.
  • The new slowest test is 11.5 s (the wiki permissions crawl); none of the top 15 are network waits.

🤖 Generated with Claude Code

Closes #937. The default backend run goes from **1,061 s to 536 s** locally, with the same tests passing. **1. Stub the local-Whisper `/health` probe by default.** `LocalWhisperProvider.submit()` probes `/health` before transcribing. Tests mocked the transcribe call but not the probe, so about 30 recording-pipeline tests each paid a failing DNS lookup (~3 s) plus the 5 s `_HEALTH_TIMEOUT`, then silently fell back to v1. The fixture that clears the capability cache (#352) meant every test paid it again. An autouse fixture now returns the probe's own fallback value directly. Tests that exercise the probe carry `@pytest.mark.real_asr_probe` and mock the HTTP themselves: two whole modules, plus individual tests in six others. **2. A network guard.** An autouse fixture wraps `socket.getaddrinfo`. Resolving anything other than loopback, an IP literal or the test database's host raises `gaierror` immediately, and the test **fails at teardown naming the host**, even when the code under test swallowed the error, as the probe did. The opt-out is `@pytest.mark.allow_network`; no test needed it. It caught one real case: `test_a_hostname_that_does_not_resolve_says_so` depended on live DNS and now mocks the failure. There's no new dependency, because `pytest-socket` guards `connect()`, and the lookup is where most of the time went. **3. Synthetic-session harness out of the default run.** The four `test_synthetic_session_*` modules are marked `synthetic`, and the backend CI job runs `pytest -x -q -m "not synthetic"`. The `synthetic-session` job is unchanged: it passes explicit files and there's no `addopts`, so it still runs all 105 tests. They stay serial, per #684. Also updated: the `/qb-test` recipe (backend deselects `synthetic`, plus a new `synthetic` section) and a changelog fragment. **Verified locally:** - Default run: 5769 passed, 14 skipped, 105 deselected in 536 s. - `-m synthetic`: 105 passed in 294 s. - ruff 0.4.4 format and check: clean. - The new slowest test is 11.5 s (the wiki permissions crawl); none of the top 15 are network waits. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
test(repo): synthetic-session harness out of the default backend run (#937)
All checks were successful
CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 1m34s
Details
CI / Does this change touch the recording path? (pull_request) Successful in 17s
Details
CI / Bot/backend version sync (pull_request) Successful in 25s
Details
CI / Backend lint (ruff) (pull_request) Successful in 32s
Details
CI / Feedback site tests (pull_request) Successful in 42s
Details
CI / Bot tests and audit (pull_request) Successful in 2m43s
Details
CI / Docker image build (pull_request) Successful in 4m20s
Details
CI / Synthetic session harness (no GPU, no LLM) (pull_request) Successful in 12m46s
Details
CI / Backend migration, tests, and audit (pull_request) Successful in 27m58s
Details
CI / Frontend tests, audit, and build (pull_request) Successful in 29m45s
Details
CI / End-to-end journeys (desktop, phone, keyboard-only) (pull_request) Successful in 25m54s
Details
cb7e78260d 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
claude-bot scheduled this pull request to auto merge when all checks succeed 2026-09-29 17:16:30 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Issue related to the backend

  
bot
Issue related to the bot

  
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
frontend
Issue related to the frontend

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
recording
Issue related to the recording system

  
security
Security hardening / vulnerability work

  
wontfix
This won't be fixed

No labels
backend
bot
bug
duplicate
enhancement
frontend
help wanted
invalid
question
recording
security
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/Quest-Board!938
No description provided.