Add worker and job lifecycle tests

claude-bot commented

2026-08-02 07:17:40 +00:00

Author

Done in 44a3455.

Most of this was already covered

Landed with #2, #3, #4 and #82, in test_worker_runtime.py (31 tests), test_ai_handler.py (28), test_ocr_handler.py (32) and test_evidence_chain_and_reruns.py:

full lifecycle queued → running → success, with started_at, finished_at, result_summary and the claim released
failure → retry_count increments → requeued → exhaustion → failed with a readable error_message, plus PermanentJobError not retried
a handler that raises mid-write commits nothing (no orphan evidence)
two workers racing one job: exactly one execution; N jobs run exactly once each
duplicate enqueue idempotent, including the true constraint collision (#82)
reruns append for both AI and OCR, with the earlier rows still active
lease reclamation, backoff, and a worker that lost its claim not writing its result

Two things were genuinely missing.

1. "No test performs a real network call" was a docstring

Which is true until someone adds a test that reaches for the real thing. Not hypothetical here: this codebase calls a paid vision API, so an accidentally-live test bills money on every CI run and turns a provider outage into a red build.

It is now an autouse fixture that refuses outbound connect and connect_ex, allowing loopback, with @pytest.mark.network as the opt-in that the two live-API suites carry (they were already skipped without credentials — the marker records the intent rather than granting it).

Two choices worth naming:

At the socket, not per client. The guarantee is about the process, not about the one HTTP library we remembered to mock. A future dependency that brings its own client is covered without anyone doing anything.
connect_ex too. It returns an error code rather than raising, so a client using it would have walked straight past a guard on connect alone.

test_harness_isolation.py asserts the guard works, that loopback still does, and that the marker really opts out.

2. `JobStatus.cancelled` existed and nothing set it

No route, no CLI, no repository method — it has been in the enum since the first migration with neither a producer nor a consumer. That is a fair state for Phase 1 to be in. What is not fair is the worker being undefined against a status that already exists in the schema, and which an operator can set by hand today to stop a job that is misbehaving.

Three guarantees pinned:

A cancelled job is never claimed.
The lease sweep does not resurrect one. Requeueing a cancelled job would undo the cancel with nothing anywhere showing why the job started again — the most confusing possible outcome.
A job cancelled while running has its result discarded. This is the half that matters. Stopping a job from starting is easy; the case that bites is the one already in flight, where the handler finishes after the cancel and would otherwise write success over it — leaving a job that was cancelled, reported as completed, with evidence in the archive to match.

All three hold today, for load-bearing reasons rather than by accident: the claim is UPDATE ... WHERE status = 'queued', and _finalize re-checks both owner and status before writing. Which is precisely why they were worth stating before something depends on them.

Startup recovery also now runs through run_forever rather than calling reclaim_lapsed() by hand — "recovered at startup" is the claim, and calling the sweep directly would prove only that the sweep works.

Done when

Success, failure, retry-exhaustion, and cancellation paths are covered
Concurrent claim safety is proven
Duplicate enqueue is idempotent
Reruns are proven append-only for both AI and OCR
No test performs a real network call — now enforced, not asserted

965 passed, 8 skipped; ruff clean.

Done in 44a3455. ## Most of this was already covered Landed with #2, #3, #4 and #82, in `test_worker_runtime.py` (31 tests), `test_ai_handler.py` (28), `test_ocr_handler.py` (32) and `test_evidence_chain_and_reruns.py`: - full lifecycle `queued → running → success`, with `started_at`, `finished_at`, `result_summary` and the claim released - failure → `retry_count` increments → requeued → exhaustion → `failed` with a readable `error_message`, plus `PermanentJobError` not retried - a handler that raises mid-write commits nothing (no orphan evidence) - two workers racing one job: exactly one execution; N jobs run exactly once each - duplicate enqueue idempotent, including the true constraint collision (#82) - reruns append for both AI and OCR, with the earlier rows still active - lease reclamation, backoff, and a worker that lost its claim not writing its result Two things were genuinely missing. ## 1. "No test performs a real network call" was a docstring Which is true until someone adds a test that reaches for the real thing. Not hypothetical here: this codebase calls a **paid** vision API, so an accidentally-live test bills money on every CI run and turns a provider outage into a red build. It is now an autouse fixture that refuses outbound `connect` and `connect_ex`, allowing loopback, with `@pytest.mark.network` as the opt-in that the two live-API suites carry (they were already skipped without credentials — the marker records the intent rather than granting it). Two choices worth naming: - **At the socket, not per client.** The guarantee is about the process, not about the one HTTP library we remembered to mock. A future dependency that brings its own client is covered without anyone doing anything. - **`connect_ex` too.** It returns an error code rather than raising, so a client using it would have walked straight past a guard on `connect` alone. `test_harness_isolation.py` asserts the guard works, that loopback still does, and that the marker really opts out. ## 2. `JobStatus.cancelled` existed and nothing set it No route, no CLI, no repository method — it has been in the enum since the first migration with neither a producer nor a consumer. That is a fair state for Phase 1 to be in. What is not fair is the **worker being undefined against a status that already exists in the schema**, and which an operator can set by hand today to stop a job that is misbehaving. Three guarantees pinned: 1. **A cancelled job is never claimed.** 2. **The lease sweep does not resurrect one.** Requeueing a cancelled job would undo the cancel with nothing anywhere showing why the job started again — the most confusing possible outcome. 3. **A job cancelled *while running* has its result discarded.** This is the half that matters. Stopping a job from starting is easy; the case that bites is the one already in flight, where the handler finishes after the cancel and would otherwise write `success` over it — leaving a job that was cancelled, reported as completed, with evidence in the archive to match. All three hold today, for load-bearing reasons rather than by accident: the claim is `UPDATE ... WHERE status = 'queued'`, and `_finalize` re-checks both owner and status before writing. Which is precisely why they were worth stating before something depends on them. Startup recovery also now runs through `run_forever` rather than calling `reclaim_lapsed()` by hand — "recovered at startup" is the claim, and calling the sweep directly would prove only that the sweep works. ## Done when - [x] Success, failure, retry-exhaustion, and cancellation paths are covered - [x] Concurrent claim safety is proven - [x] Duplicate enqueue is idempotent - [x] Reruns are proven append-only for both AI and OCR - [x] No test performs a real network call — now enforced, not asserted **965 passed, 8 skipped**; ruff clean.

2026-07-28 04:52:51 +00:00

Context

The worker runtime, AI handler, and OCR handler are new code in this milestone
handling durable state and external API calls. They need their own coverage.

Scope

Tests for job lifecycle, retry behaviour, idempotency, and append-only evidence.

Implementation notes

Full lifecycle: queued -> running -> success, with started_at and
finished_at stamped and result_summary written.
Failure path: handler raises, retry_count increments, job returns to queued,
and after max_retries it lands in failed with error_message set.
A crashed handler commits nothing â€” assert no orphan evidence rows.
Concurrency: two workers racing for one queued job, exactly one claim succeeds.
Idempotency: enqueuing twice with the same idempotency_key returns the existing
job instead of raising a unique-constraint 500.
Append-only: run the AI handler twice against one photo and assert two active
evidence rows exist and the first is untouched. Same for OCR.
Stub the AI and OCR backends â€” these tests must never make a network call or need
Tesseract installed.
Orphan recovery: a job left running from a previous process is requeued or failed
on startup, not stranded.

Success, failure, retry-exhaustion, and cancellation paths are covered
Concurrent claim safety is proven
Duplicate enqueue is idempotent
Reruns are proven append-only for both AI and OCR
No test performs a real network call

References

backend/app/workers/
backend/app/models/models.py (BackgroundJob)

Depends on: the background worker runtime, the AI job handler, the OCR job handler, and the backend test harness.

## Context The worker runtime, AI handler, and OCR handler are new code in this milestone handling durable state and external API calls. They need their own coverage. ## Scope Tests for job lifecycle, retry behaviour, idempotency, and append-only evidence. ## Implementation notes - Full lifecycle: `queued` -> `running` -> `success`, with `started_at` and `finished_at` stamped and `result_summary` written. - Failure path: handler raises, `retry_count` increments, job returns to `queued`, and after `max_retries` it lands in `failed` with `error_message` set. - A crashed handler commits nothing â€” assert no orphan evidence rows. - Concurrency: two workers racing for one queued job, exactly one claim succeeds. - Idempotency: enqueuing twice with the same `idempotency_key` returns the existing job instead of raising a unique-constraint 500. - Append-only: run the AI handler twice against one photo and assert two active evidence rows exist and the first is untouched. Same for OCR. - Stub the AI and OCR backends â€” these tests must never make a network call or need Tesseract installed. - Orphan recovery: a job left `running` from a previous process is requeued or failed on startup, not stranded. ## Done when - [ ] Success, failure, retry-exhaustion, and cancellation paths are covered - [ ] Concurrent claim safety is proven - [ ] Duplicate enqueue is idempotent - [ ] Reruns are proven append-only for both AI and OCR - [ ] No test performs a real network call ## References - `backend/app/workers/` - `backend/app/models/models.py` (`BackgroundJob`) Depends on: the background worker runtime, the AI job handler, the OCR job handler, and the backend test harness.

claude-bot added this to the v0.2.0 milestone

claude-bot added the

testing

worker

backend

labels

rbrooks referenced this issue from a commit

2026-08-02 07:17:04 +00:00

Close the network to the test suite, and define cancellation

rbrooks closed this issue