Verify the AI dating path against the live Anthropic API #136

New issue

Open

opened 2026-08-03 18:34:49 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-08-03 18:34:49 +00:00

Context

#3 built the whole AI dating path — ClaudeVisionBackend, the structured-output schema, the Pydantic semantic validation, the api_usage_log cost ledger, the daily/monthly budget ceilings, and the ai_analysis job handler. All of it is covered by tests, but every one of those tests uses a mocked SDK.

TestAgainstTheRealAPI in backend/tests/test_ai_backend.py is the one place a real, billable call would be made, and it skips itself when no key is configured. No key has been configured, so the live path has never run.

That is a deliberate position rather than an oversight: there is no Anthropic key set up for this project yet, and the rest of the work was worth finishing first. This issue exists so the gap is recorded rather than remembered.

What has and has not been proven

Proven, against a stub: the job lifecycle, evidence writing, reliability mapping from confidence, the budget check running before the call, the spend ledger recording failures and refusals as well as successes, permanent-vs-retryable failure classification, image downscaling through the #65 sandbox, and that a malformed estimate never reaches a date column.

Not proven, because it needs the real service:

the request shape is one the API accepts — model id, output_config.format schema, image block encoding, max_tokens
the response parses: stop_reason, the structured-output payload, and the refusal path
token counts and therefore cost arithmetic — api_usage_log.cost_usd is computed from CIRCA_AI_INPUT_PRICE_PER_MTOK / CIRCA_AI_OUTPUT_PRICE_PER_MTOK against usage numbers no real response has ever supplied
the SDK's typed exceptions map to the failure kinds the handler distinguishes

The third is the one with teeth: the budget ceilings (§5.5, $5/day and $30/month) are a sum over api_usage_log, so if the cost arithmetic is wrong the ceiling is wrong, and it fails in the expensive direction without saying anything.

How to verify

cd backend
CIRCA_AI_API_KEY=sk-... .venv/bin/pytest tests/test_ai_backend.py -k RealAPI

Then end to end through the worker, which is what actually exercises the ledger:

CIRCA_AI_API_KEY=sk-... CIRCA_AI_BACKEND=auto \
  .venv/bin/python -m app.workers.runner --once

Done when

TestAgainstTheRealAPI runs (not skips) and passes
One ai_analysis job completes end to end against the real API
api_usage_log holds a row whose input_tokens / output_tokens / cost_usd are checked by hand against the provider's own reported usage for that call
A refusal and a rate-limit response are each observed, or a note records why they could not be provoked

References

backend/app/services/ai_claude.py, ai_budget.py
backend/app/workers/handlers/ai_analysis.py
backend/tests/test_ai_backend.py (TestAgainstTheRealAPI, marked @pytest.mark.network)
Migration 009_api_usage_log.py
docs/circa-spec.md §5.5 (budgets), §10.1 (reliability)

Deferred until a key is set up and live deployment is in scope. Nothing else depends on it: CIRCA_AI_BACKEND=mock exercises the whole path for free, and auto refuses AI jobs with a clear message when no key is present.

## Context #3 built the whole AI dating path — `ClaudeVisionBackend`, the structured-output schema, the Pydantic semantic validation, the `api_usage_log` cost ledger, the daily/monthly budget ceilings, and the `ai_analysis` job handler. All of it is covered by tests, but **every one of those tests uses a mocked SDK**. `TestAgainstTheRealAPI` in `backend/tests/test_ai_backend.py` is the one place a real, billable call would be made, and it skips itself when no key is configured. No key has been configured, so **the live path has never run**. That is a deliberate position rather than an oversight: there is no Anthropic key set up for this project yet, and the rest of the work was worth finishing first. This issue exists so the gap is recorded rather than remembered. ## What has and has not been proven **Proven, against a stub:** the job lifecycle, evidence writing, reliability mapping from confidence, the budget check running *before* the call, the spend ledger recording failures and refusals as well as successes, permanent-vs-retryable failure classification, image downscaling through the #65 sandbox, and that a malformed estimate never reaches a date column. **Not proven, because it needs the real service:** - the request shape is one the API accepts — model id, `output_config.format` schema, image block encoding, `max_tokens` - the response parses: `stop_reason`, the structured-output payload, and the refusal path - token counts and therefore **cost arithmetic** — `api_usage_log.cost_usd` is computed from `CIRCA_AI_INPUT_PRICE_PER_MTOK` / `CIRCA_AI_OUTPUT_PRICE_PER_MTOK` against usage numbers no real response has ever supplied - the SDK's typed exceptions map to the failure kinds the handler distinguishes The third is the one with teeth: the budget ceilings (§5.5, $5/day and $30/month) are a sum over `api_usage_log`, so if the cost arithmetic is wrong the ceiling is wrong, and it fails in the expensive direction without saying anything. ## How to verify ```bash cd backend CIRCA_AI_API_KEY=sk-... .venv/bin/pytest tests/test_ai_backend.py -k RealAPI ``` Then end to end through the worker, which is what actually exercises the ledger: ```bash CIRCA_AI_API_KEY=sk-... CIRCA_AI_BACKEND=auto \ .venv/bin/python -m app.workers.runner --once ``` ## Done when - [ ] `TestAgainstTheRealAPI` runs (not skips) and passes - [ ] One `ai_analysis` job completes end to end against the real API - [ ] `api_usage_log` holds a row whose `input_tokens` / `output_tokens` / `cost_usd` are checked by hand against the provider's own reported usage for that call - [ ] A refusal and a rate-limit response are each observed, or a note records why they could not be provoked ## References - `backend/app/services/ai_claude.py`, `ai_budget.py` - `backend/app/workers/handlers/ai_analysis.py` - `backend/tests/test_ai_backend.py` (`TestAgainstTheRealAPI`, marked `@pytest.mark.network`) - Migration `009_api_usage_log.py` - `docs/circa-spec.md` §5.5 (budgets), §10.1 (reliability) Deferred until a key is set up and live deployment is in scope. Nothing else depends on it: `CIRCA_AI_BACKEND=mock` exercises the whole path for free, and `auto` refuses AI jobs with a clear message when no key is present.

claude-bot added this to the v1.0.0 milestone

2026-08-03 18:34:57 +00:00

claude-bot added the

backend

testing

labels

2026-08-03 18:35:05 +00:00