Stop the harness guards sleeping through the retry backoff (#684)

claude-bot commented

2026-09-17 20:56:25 +00:00

Collaborator

Refs #684. One autouse fixture. No production code, no new dependency.

What was actually slow

Profiling the job rather than reasoning about it, the guards file was 17m29s of a ~28-minute job, and within it one test was 127s of 271s:

127.29s test_a_server_error_on_every_span_of_one_track_fails_loudly
 36.30s test_a_server_that_permutes_its_echoed_labels_changes_nothing
 35.57s setup (the assembled session fixture)
 19.20s test_a_skipped_speaker_below_the_threshold_only_warns
 …the remaining four are under 3s each

That test makes every span of one track return 500. corvin has 38 spans, _SPAN_ATTEMPTS is 3, and _transcribe_span_resiliently does await asyncio.sleep(_SPAN_RETRY_BACKOFF_SECONDS * attempt) between attempts — 1s then 2s. That is 114 seconds of asyncio.sleep in a test measured at 127.

The backoff is kindness to a struggling ASR server. Everything in this file talks to an in-process shim that is not struggling and has no rate limit.

The change

An autouse fixture sets the backoff to zero for this file only. The attempt count is deliberately untouched, so what the guard asserts is unchanged: every attempt is still made and still fails, the span is still skipped, the failure share still passes _SPAN_MAX_FAILURE_SHARE, and the run still raises.

	before	after
guards file	271s	152s
that one test	127.29s	16.48s

manifest + scorers unchanged at 74 passed / 17.9s, confirming the fixture is scoped to this file and has not leaked.

pytest-xdist was tried and rejected — with numbers

I proposed this first in #684. It does not hold up:

	12 CPU	2 CPU
serial (with this fix)	166s	156s
`-n 2`	137s	158s
`-n 4`	132s	236s
`-n 6`	135s	—

Serial costs the same at 2 cores as at 12, because the work is ffmpeg process-spawn latency and serialised I/O, not CPU. So parallelism buys little, and it pays a per-worker copy of the ~36s module-scoped session fixture: -n 4 is 20% faster on a wide machine and 50% slower on a narrow one. The runner's width isn't known here, so that is a regression risk for a small gain.

Left serial, with the measurements written into ci.yml so the next person doesn't repeat the experiment.

What this does not fix

The ffmpeg spawn cost in #684 is real and still there — ~232 spawns per VAD transcribe, and the same path runs in production on every session. That wants the segment-muxer change and a reworked timeout, as its own piece of work with before/after timings. This PR is the free half.

Also worth noting separately: the serial guards run took 271s on a dev laptop vs 1,049s on the runner — the runner is roughly 4× slower, which is its own question.

🤖 Generated with Claude Code

Refs #684. One autouse fixture. No production code, no new dependency. ## What was actually slow Profiling the job rather than reasoning about it, the guards file was 17m29s of a ~28-minute job, and within it **one test was 127s of 271s**: ``` 127.29s test_a_server_error_on_every_span_of_one_track_fails_loudly 36.30s test_a_server_that_permutes_its_echoed_labels_changes_nothing 35.57s setup (the assembled session fixture) 19.20s test_a_skipped_speaker_below_the_threshold_only_warns …the remaining four are under 3s each ``` That test makes **every span of one track return 500**. `corvin` has 38 spans, `_SPAN_ATTEMPTS` is 3, and `_transcribe_span_resiliently` does `await asyncio.sleep(_SPAN_RETRY_BACKOFF_SECONDS * attempt)` between attempts — 1s then 2s. That is **114 seconds of `asyncio.sleep`** in a test measured at 127. The backoff is kindness to a struggling ASR server. Everything in this file talks to an in-process shim that is not struggling and has no rate limit. ## The change An autouse fixture sets the backoff to zero **for this file only**. The attempt *count* is deliberately untouched, so what the guard asserts is unchanged: every attempt is still made and still fails, the span is still skipped, the failure share still passes `_SPAN_MAX_FAILURE_SHARE`, and the run still raises. | | before | after | |---|---:|---:| | guards file | 271s | **152s** | | that one test | 127.29s | **16.48s** | `manifest` + `scorers` unchanged at 74 passed / 17.9s, confirming the fixture is scoped to this file and has not leaked. ## pytest-xdist was tried and rejected — with numbers I proposed this first in #684. It does not hold up: | | 12 CPU | 2 CPU | |---|---:|---:| | serial (with this fix) | 166s | 156s | | `-n 2` | 137s | 158s | | `-n 4` | 132s | **236s** | | `-n 6` | 135s | — | Serial costs the same at 2 cores as at 12, because the work is ffmpeg **process-spawn latency and serialised I/O**, not CPU. So parallelism buys little, and it pays a per-worker copy of the ~36s module-scoped session fixture: `-n 4` is 20% faster on a wide machine and **50% slower on a narrow one**. The runner's width isn't known here, so that is a regression risk for a small gain. Left serial, with the measurements written into `ci.yml` so the next person doesn't repeat the experiment. ## What this does not fix The ffmpeg spawn cost in #684 is real and still there — ~232 spawns per VAD transcribe, and the same path runs in production on every session. That wants the `segment`-muxer change and a reworked timeout, as its own piece of work with before/after timings. This PR is the free half. Also worth noting separately: the serial guards run took **271s on a dev laptop vs 1,049s on the runner** — the runner is roughly 4× slower, which is its own question. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit

2026-09-17 20:56:25 +00:00

perf(ci): stop the harness guards sleeping through the retry backoff (#684 )

CI / Docker image build (pull_request) Successful in 40s

Details

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 50s

Details

CI / Bot/backend version sync (pull_request) Successful in 1m22s

Details

CI / Backend lint (ruff) (pull_request) Successful in 1m27s

Details

CI / Feedback site tests (pull_request) Successful in 1m49s

Details

CI / Bot tests and audit (pull_request) Successful in 1m56s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 2m27s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 8m34s

Details

CI / Synthetic session harness (no GPU, no LLM) (pull_request) Successful in 21m39s

Details

b4e1c3a5ff

The synthetic session harness had grown to 20-30 minutes a run and was the
long pole on every PR. Profiling the job rather than reasoning about it:

  Manifest, scene and scorers   74 tests   1m23s
  Assembly and fingerprints     21 tests   7m42s
  Guards and negative fixtures  10 tests  17m29s

and within the guards, one test was 127s of 271s locally:

  127.29s test_a_server_error_on_every_span_of_one_track_fails_loudly
   36.30s test_a_server_that_permutes_its_echoed_labels_changes_nothing
   35.57s setup (the assembled session fixture)
   ...

That test makes every span of one track return 500. corvin has 38 spans,
_SPAN_ATTEMPTS is 3, and _transcribe_span_resiliently sleeps
_SPAN_RETRY_BACKOFF_SECONDS * attempt between them -- 1s then 2s, so 114
seconds of asyncio.sleep. The backoff is kindness to a struggling ASR
server; everything in this file talks to an in-process shim that is not
struggling and has no rate limit.

An autouse fixture sets the backoff to zero for this file. The attempt
*count* is deliberately left alone, so what the guard asserts is unchanged:
every attempt is still made and still fails, the span is still skipped, the
share still passes _SPAN_MAX_FAILURE_SHARE, and the run still raises.

  guards, before: 271s      that test: 127.29s
  guards, after:  152s      that test:  16.48s

74 manifest/scorer tests unchanged at 17.9s, confirming the fixture is
scoped to this file and has not leaked.

## pytest-xdist was tried and rejected, with numbers

                        12 cpu    2 cpu
  serial (fixed)         166s     156s
  -n 2                   137s     158s
  -n 4                   132s     236s
  -n 6                   135s       --

Serial costs the same at 2 cores as at 12, because the work is ffmpeg
process-spawn latency and serialised I/O rather than CPU. So parallelism
buys little, and it pays a per-worker copy of the ~36s module-scoped
session fixture: -n 4 is 20% faster on a wide machine and 50% *slower* on a
narrow one. Since the runner's width is not known here, that is a
regression risk for a small gain. Left serial, with the measurements
recorded in ci.yml so the next person does not re-run the experiment.

Refs #684.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

rbrooks referenced this pull request from a commit

2026-09-17 21:23:04 +00:00

Merge PR #685: stop the harness guards sleeping through the retry backoff (#684)

claude-bot commented

2026-09-17 21:23:09 +00:00

Author

Collaborator

Merged to main in 2b913ef.

Confirmed on CI, not just locally: the "Guards and negative fixtures" step went 17m29s → 11m59s (run #9871 vs #9846). That is −31%, against −44% locally — the difference being that the non-sleeping parts of the file are proportionally slower on the runner.

Closing by hand; the API merge returns 422 on this repo even with CI complete and mergeable: true.

Merged to `main` in 2b913ef. **Confirmed on CI**, not just locally: the "Guards and negative fixtures" step went **17m29s → 11m59s** (run [#9871](https://git.rhoving.com/rbrooks/Quest-Board/actions/runs/1153) vs [#9846](https://git.rhoving.com/rbrooks/Quest-Board/actions/runs/1149)). That is −31%, against −44% locally — the difference being that the non-sleeping parts of the file are proportionally slower on the runner. Closing by hand; the API merge returns 422 on this repo even with CI complete and `mergeable: true`.

claude-bot closed this pull request

2026-09-17 21:23:12 +00:00

CI / Docker image build (pull_request) Successful in 40s

Required

Details

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 50s

Required

Details

CI / Bot/backend version sync (pull_request) Successful in 1m22s

Required

Details

CI / Backend lint (ruff) (pull_request) Successful in 1m27s

Required

Details

CI / Feedback site tests (pull_request) Successful in 1m49s

Required

Details

CI / Bot tests and audit (pull_request) Successful in 1m56s

Required

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 2m27s

Required

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 8m34s

Required

Details

CI / Synthetic session harness (no GPU, no LLM) (pull_request) Successful in 21m39s

Required

Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Rows
Columns

Stop the harness guards sleeping through the retry backoff (#684) #685

What was actually slow

The change

pytest-xdist was tried and rejected — with numbers

What this does not fix

Pull request closed