Implement the AI analysis job handler and pluggable Claude vision backend #3
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
JobType.ai_analysiscan be enqueued andEvidenceSource.aiexists in the schema,but there is no AI backend in the codebase — no client, no prompt, no cost control.
The product spec names the Claude API as the default AI backend with cost controls.
Scope
A pluggable AI backend interface plus a Claude vision implementation, and the
ai_analysisjob handler that turns a model response into an appendedDateEvidencerow.Implementation notes
AIBackendas a protocol/ABC so the Phase 4 local ollama/llava backendcan drop in without touching the handler. Select via config
(
CIRCA_AI_BACKEND,CIRCA_AI_API_KEY,CIRCA_AI_MODEL).DatePrecision,a confidence in 0-1, and a short rationale. Request structured JSON output and
validate it with Pydantic before it touches the DB — never trust free text into
a date column.
DateEvidence:source = ai,reliabilityper the spec'strust rules,
date_low/date_high/precision,confidence, rationale intonotes, andjob_idset to the job that produced it.that rule is called out as a Phase 1 risk in the plan. A second rerun produces a
second row, and the evidence panel shows both.
refusal path when the budget is exhausted (job fails with a clear message rather
than silently skipping). Record per-job cost in
result_summaryso the Phase 4cost dashboard has data to read.
refusal. Each should produce a retryable failure with a useful
error_message.Done when
References
backend/app/models/models.py(DateEvidence,EvidenceSource.ai,JobType.ai_analysis)backend/app/api/routes/jobs.pydocs/circa-spec.md(AI backend, cost controls, handwritten-date trust)docs/circa-phase1-plan.mdsection 11.5Depends on: the background worker runtime.
Amended by the audit of 2026-07-28.
Two additions:
docs/circa-spec.md§8.2 says low AI confidence should triggerthe rescan flag automatically. Given the physical originals are degrading, identifying scans worth
redoing soon has real time value.
no pixel cap, no
verify(), and no isolation. Moving image work into this worker underRLIMIT_AS/RLIMIT_CPUmeans a decoder bug or decompression bomb cannot take down the APIprocess, which holds the session-signing key.
Also note #112 (format-rule evidence) wants
photo_format_cluesfrom this handler's structuredoutput, so include that field in the response schema.
Picking this up now that #2 (worker runtime) and #4 (OCR) are done. Recording the decisions before implementing, as with #4.
Client: the official
anthropicSDK, not raw HTTP. It carries the request/response types, typed exceptions, and retry behaviour; hand-rolling those against a moving API is how a client silently drifts from the service it talks to. Added topyproject.toml; both lockfiles regenerated withpip-compilein a Python 3.12 container to match how they were originally built (only theanthropicchain was added —distro,docstring-parser,jiter,sniffio).Model:
claude-opus-5, configurable. Priced at $5/$25 per MTok, which is what the cost calculation uses — the per-MTok rates are config, not constants, so changing the model cannot silently mis-bill.Budgets: the spec's numbers (§5.5) — $5/day, $30/month, enforced before each run, job fails with an explicit message when exceeded rather than silently skipping. §14 estimates ~$30 total for 10,000 photos at ~$0.003/image, so the monthly ceiling is roughly one full pass over the collection.
Structured output, validated twice. The request uses
output_config.formatwith a JSON schema so the model is constrained to the shape; the response is then validated through a Pydantic model before anything reaches the database — the issue's "never trust free text into a date column". The two are not redundant: the schema constrains shape, Pydantic enforces semantics the JSON-schema subset cannot express (year_range_low <= year_range_high, both inside the collection's plausible window, confidence in 0–1,year_estimateinside its own range). A response that violates any of those is rejected before the write and the job retries.Response schema includes
photo_format_clues, per the amendment above, so #112 has the field it wants.Reliability follows confidence, per spec §10.1 ("AI visual estimate — Low-Medium, depends on confidence"): above a configurable threshold the evidence is
medium, below itlow. That distinction is load-bearing rather than cosmetic —mediumpromotes apendingphoto toneeds_reviewunderprojections.fold()andlowdoes not, so a confident estimate queues the photo for a reviewer and a hesitant one only sits in the evidence panel.One thing I have to flag: there is no
ANTHROPIC_API_KEYin the development environment, so I cannot make a single real call. The backend is covered by tests against a mocked SDK (request shape, refusal handling, error mapping, cost arithmetic, budget refusal, validation rejection), and a live test that skips without a key — the same pattern #4 used for tesseract. The first real call will be yours, and the sensible first move is one photo with--once, not a batch.Also implementing
APIUsageLog(spec §12), which the schema does not have yet — migration009. Without it "check daily and monthly limits before each batch" has nothing to read.Done in
53ce3d0. CI green.Done when
59 tests across
test_ai_backend.pyandtest_ai_handler.py; full suite 670 passed.⚠️ Not verified against the real API. There is no key in this environment, so nothing here has made a single billable call. The client is tested against a stub SDK — refusal, rate limit, auth failure, malformed answer, cost arithmetic — and
TestAgainstTheRealAPIskips without a key. Before running a batch, run one photo:then check the estimate on that photo and the row in
api_usage_log. If the request shape is wrong, that is where it will show, and one photo is a cheap way to find out.Validation is doubled, deliberately.
output_config.formatconstrains the shape;DateEstimateenforces the semantics the structured-output schema subset cannot express.{"year_range_low": 1990, "year_range_high": 1970}satisfies the schema and is impossible — as does a confidence of 1.5, a year of 1700, and a point estimate outside its own range. Nine such cases are tested, and there is a test asserting the schema carries nominimum/maximum, since adding them would fail the request outright and look like a fix.The ledger records failures too. A refused or errored call still consumed input tokens. A ledger that recorded only successes would understate spend at exactly the moment something is going wrong — and the budget check is a
SUMover that ledger, so an unrecorded call is invisible to the next caller's ceiling.Budget exhaustion is permanent, not retried. The ceiling will not move until someone raises it or the month turns, so a retrying job would spend its whole budget re-reading the same limit and end with a less informative message than the one it started with.
One thing to keep in step:
CIRCA_AI_INPUT_PRICE_PER_MTOK/..._OUTPUT_...are what the cost is computed from. ChangingCIRCA_AI_MODELwithout changing them means spending against a budget calculated from the wrong numbers. They are configuration precisely so a model swap cannot silently mis-bill, but that does cut both ways.Two things this needed along the way
api_usage_log(migration009) — spec §12 specifies it and the schema did not have it. Without it "check daily and monthly limits" has nothing to read. It also gives #723 (cost dashboard) its data source.downscaleoperation in the #65 sandbox. Scans are routinely larger than the API accepts, and image tokens dominate per-photo cost. Writing it, the existing boundary test caught me importingapp.configinto the module the sandbox child re-imports — which would have parsed settings in every sandboxed process. The parent-side helper moved toimage_prep.py; the sandbox module stays free ofappimports and the test still enforces it.Following on
photo_format_clueson every AI evidence row, as the amendment above asked.