Backend test suite: stop tests waiting on the real network, block unmocked network access, and move the synthetic harness out of the default run #937

New issue

Closed

opened 2026-09-29 16:41:25 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-09-29 16:41:25 +00:00

Collaborator

Measured 2026-09-29. A full local backend run takes about 17.7 min (1,061 s, 5,878 tests). Where the time goes (junit timings plus cProfile of individual tests):

About 5,730 tests are fast (under 0.5 s each, ~320 s in total).
About 30 recording-pipeline tests cluster at almost exactly 8 s or 5 s. LocalWhisperProvider.submit() calls discover_capabilities() → probe_local_whisper_capabilities(), which GETs /health over the real network. The tests mock the transcribe call but not the probe, so every such test pays a failing DNS lookup (~3 s in the container) plus _HEALTH_TIMEOUT = 5.0, then silently falls back to v1, so it still passes. The per-endpoint cache is deliberately cleared around every test (reset_asr_capability_cache, #352), so every test pays it again. A cProfile of two such tests: 6 s in getaddrinfo, 10 s idle in epoll, out of 27 s.
The synthetic-session harness takes ~280 s over 31 tests (the assembly fixture alone takes 75 s). The dedicated CI job synthetic-session already runs these files (gated on audio paths, and always on main; deliberately serial per #684), but a plain pytest collects them as well.

Plan:

Stub the ASR /health probe by default with an autouse fixture that returns the default capabilities with no I/O. Tests that exercise the probe itself opt out with a real_asr_probe marker.
Add a network guard in conftest: resolving any host other than loopback or the test database's host fails immediately and loudly, naming the host. This is a fixture, not a new plugin. pytest-socket guards connect(), but the DNS lookup is where most of the wait is. Tests that need a genuinely local server opt out with an allow_network marker. Fix what it surfaces by mocking, not by allowlisting.
Mark the synthetic-session modules synthetic. The backend CI job and the local /qb-test backend recipe run -m "not synthetic". The synthetic-session job is unchanged.

Done when: the full default run is green, with before and after timings recorded here, and the synthetic-session job still runs the same tests.

**Measured 2026-09-29.** A full local backend run takes about 17.7 min (1,061 s, 5,878 tests). Where the time goes (junit timings plus cProfile of individual tests): - About 5,730 tests are fast (under 0.5 s each, ~320 s in total). - **About 30 recording-pipeline tests cluster at almost exactly 8 s or 5 s.** `LocalWhisperProvider.submit()` calls `discover_capabilities()` → `probe_local_whisper_capabilities()`, which GETs `/health` over the **real network**. The tests mock the transcribe call but not the probe, so every such test pays a failing DNS lookup (~3 s in the container) plus `_HEALTH_TIMEOUT = 5.0`, then silently falls back to v1, so it still passes. The per-endpoint cache is deliberately cleared around every test (`reset_asr_capability_cache`, #352), so every test pays it again. A cProfile of two such tests: 6 s in `getaddrinfo`, 10 s idle in epoll, out of 27 s. - **The synthetic-session harness takes ~280 s over 31 tests** (the assembly fixture alone takes 75 s). The dedicated CI job `synthetic-session` already runs these files (gated on audio paths, and always on main; deliberately serial per #684), but a plain `pytest` collects them as well. **Plan:** 1. Stub the ASR `/health` probe by default with an autouse fixture that returns the default capabilities with no I/O. Tests that exercise the probe itself opt out with a `real_asr_probe` marker. 2. Add a network guard in conftest: resolving any host other than loopback or the test database's host fails immediately and loudly, naming the host. This is a fixture, not a new plugin. `pytest-socket` guards `connect()`, but the DNS lookup is where most of the wait is. Tests that need a genuinely local server opt out with an `allow_network` marker. Fix what it surfaces by mocking, not by allowlisting. 3. Mark the synthetic-session modules `synthetic`. The backend CI job and the local `/qb-test backend` recipe run `-m "not synthetic"`. The `synthetic-session` job is unchanged. **Done when:** the full default run is green, with before and after timings recorded here, and the `synthetic-session` job still runs the same tests.

rbrooks referenced this issue from a commit

2026-09-29 17:16:16 +00:00

test(repo): stub the local-Whisper health probe by default (#937)

rbrooks referenced this issue from a commit

2026-09-29 17:16:16 +00:00

test(repo): fail any test that resolves an unmocked hostname (#937)

rbrooks referenced this issue from a commit

2026-09-29 17:16:16 +00:00

test(repo): synthetic-session harness out of the default backend run (#937)

claude-bot referenced this issue from a pull request that will close it,

2026-09-29 17:16:25 +00:00

test(repo): halve the backend suite, block unmocked network in tests (#937) #938

claude-bot commented

2026-09-29 17:16:34 +00:00

Author

Collaborator

Implemented in PR #938 (commits 424bc177, 4dff40d6, cb7e7826), set to merge when checks succeed.

Timings (local, qb-backend-test:local):

Before: 1,061 s for 5,878 tests.
After: 536 s for the default run (5769 passed, 14 skipped, 105 deselected). The 105 synthetic tests pass separately in 294 s.

What the guard caught:

One test depended on live DNS (test_a_hostname_that_does_not_resolve_says_so). It now mocks the failure.
11 tests turned out to script /health themselves. They now carry real_asr_probe.
No test needed allow_network.
IP literals pass the guard, since they involve no DNS lookup.

Follow-ups, not in this PR:

The remaining slow tests are real work, not waiting: the wiki and discussion permission crawls (~11 s each) and test_silent_tracks (~10 s each).
pytest-xdist is still possible with a database per worker, but at ~9 minutes it's worth measuring the need first.

Implemented in PR #938 (commits 424bc177, 4dff40d6, cb7e7826), set to merge when checks succeed. **Timings (local, `qb-backend-test:local`):** - Before: 1,061 s for 5,878 tests. - After: 536 s for the default run (5769 passed, 14 skipped, 105 deselected). The 105 synthetic tests pass separately in 294 s. **What the guard caught:** - One test depended on live DNS (`test_a_hostname_that_does_not_resolve_says_so`). It now mocks the failure. - 11 tests turned out to script `/health` themselves. They now carry `real_asr_probe`. - No test needed `allow_network`. - IP literals pass the guard, since they involve no DNS lookup. **Follow-ups, not in this PR:** - The remaining slow tests are real work, not waiting: the wiki and discussion permission crawls (~11 s each) and `test_silent_tracks` (~10 s each). - pytest-xdist is still possible with a database per worker, but at ~9 minutes it's worth measuring the need first.

claude-bot closed this issue

2026-09-29 17:48:26 +00:00

claude-bot referenced this issue from a commit

2026-09-29 17:48:27 +00:00

Merge PR #938: halve the backend suite, block unmocked network in tests (#937)

claude-bot referenced this issue

2026-09-29 17:58:29 +00:00

CI: frontend vitest runs on one worker, and the journeys run serially — shorten the critical path #939