#500 - feat: self-hosted first-class — a CPU profile, configurable AI timeouts, an admin hardware preflight, and measured hardware profiles (#353, #354) - rbrooks/Quest-Board

claude-bot commented

2026-09-05 04:34:46 +00:00

Collaborator

Closes #353. Closes #354.

Self-hosting without a GPU is now a supported, measured configuration rather than a hope, and a self-hoster can find out what their hardware will do before the first three-hour session rather than after it.

What

Timeouts are configuration (app/config.py, one accessor ai_timeouts()): QB_LLM_TIMEOUT_SECONDS, QB_LLM_PROSE_TIMEOUT_SECONDS, QB_ASR_TIMEOUT_SECONDS, and QB_AI_PROFILE ∈ {auto, gpu, cpu}. cpu multiplies the defaults by 6; an explicitly set value is used as written. Celery task limits derive from the timeouts (task_limits(): on gpu this reproduces #398's 6 h exactly; the invariant is that a task limit can never be shorter than the calls it contains). The admin test button and the monthly canary scale too — both could only ever report a working CPU endpoint as broken. auto follows the last preflight; the Celery limits are read at worker import, so a CPU deployment should pin cpu (stated in code, .env.example, docs and the panel). Two seams fixed on the way: #356's call_lease_ttl was handed base constants at three sites (a lease would have expired mid-call on the cpu profile), and #350's adapter had its own hardcoded 600 s.
Grammar-constrained beats on llama.cpp, proven on the wire: test_beat_schema_grammar.py drives the real extract_beats through the conformance fake and asserts the closed schema under both spellings, temperature: 0, thinking disabled. Nothing was dropping it.
Admin preflight — POST /api/admin/ai/preflight (stored under ai_preflight, GET reads it): ASR /health reachability and declared capabilities, real-time factor pooled from the last 10 session_usage rows or "not yet measured" (never a synthetic benchmark); one fixed 128-token LLM probe → tokens/s; the declared window and chunk count; the capability gaps; a throughput class by band and an expected processing time for a 3.5 h / 5-speaker session (prefill charged at 10× generation — measured ratios were 31×/11×/2.8×; validated to within ~30 % on three configurations). AdminHardwareProfile panel with a Run button.
Bands, refined from measurement: < 10 / 10–40 / > 40 tok/s, named by throughput not silicon — a GPU-less 9800X3D measures 15.6 and does not need the long timeouts, so "CPU-class" would have been wrong in both directions; only the bottom band maps to the cpu timeouts. Lower bound raised from 8 because one beat extraction on the CPU recipe (~5.7k prompt + ~0.8k completion tokens) crowds the 300 s default at 10 tok/s.

Measured (python -m evals --provider live --record, four synthetic fixtures, one run each, history.jsonl tagged 47c31d7-353-{gpu,cpu}):

	GPU: RTX 3090, qwen3.5 9B Q8, 131k window	CPU: Ryzen 9800X3D 8 threads, Qwen2.5-3B Q4_K_M, 8k window
coverage	0.850	0.500
attribution	0.725	0.833
chronology	0.975	1.000
beat validation	0.972	0.904
tok/s in pipeline	65	18
wall per 5-min fixture	30–49 s	150–186 s

Only coverage degrades on the small model: chronology and attribution come from code-side sort and validator, and beat validation holds because the schema is a decoding grammar. The CPU summary is shorter, not wronger — the issue's thesis, measured. A generic QEMU vCPU without AVX2 (the dev box) measured 0.3 tok/s and is documented as the floor no profile rescues.

Docs: OPERATIONS "Hardware profiles" — three recipes (CPU-only, modest GPU, workstation GPU) with model, hardware, timing and scores; the modest-GPU row honestly empty with the hardware and command that fills it; docs/examples/docker-compose.cpu-llm.yml as the runnable CPU-only configuration; the commands that produced every number.

Weakness stated: the private hand-verified session is not present on this machine or the dev box, so both runs used the four synthetic five-minute fixtures — neither exercised windowing, and the coverage figures describe a short transcript. ASR real-time factor is not yet measured anywhere (no session_usage rows exist yet).

Rebased over #350 and #356 (config and import conflicts, both sides kept).

Verification

Backend 2064 passed, 10 skipped; frontend 497 passed; ruff and eslint clean (one pre-existing warning); version sync OK; the CI evals gate passes. No migration.

🤖 Generated with Claude Code

Closes #353. Closes #354. Self-hosting without a GPU is now a supported, measured configuration rather than a hope, and a self-hoster can find out what their hardware will do before the first three-hour session rather than after it. ## What - **Timeouts are configuration** (`app/config.py`, one accessor `ai_timeouts()`): `QB_LLM_TIMEOUT_SECONDS`, `QB_LLM_PROSE_TIMEOUT_SECONDS`, `QB_ASR_TIMEOUT_SECONDS`, and **`QB_AI_PROFILE`** ∈ {`auto`, `gpu`, `cpu`}. `cpu` multiplies the defaults by 6; an explicitly set value is used as written. Celery task limits derive from the timeouts (`task_limits()`: on `gpu` this reproduces #398's 6 h exactly; the invariant is that a task limit can never be shorter than the calls it contains). The admin test button and the monthly canary scale too — both could only ever report a working CPU endpoint as broken. `auto` follows the last preflight; the Celery limits are read at worker import, so a CPU deployment should pin `cpu` (stated in code, `.env.example`, docs and the panel). Two seams fixed on the way: #356's `call_lease_ttl` was handed base constants at three sites (a lease would have expired mid-call on the cpu profile), and #350's adapter had its own hardcoded 600 s. - **Grammar-constrained beats on llama.cpp, proven on the wire**: `test_beat_schema_grammar.py` drives the real `extract_beats` through the conformance fake and asserts the closed schema under both spellings, `temperature: 0`, thinking disabled. Nothing was dropping it. - **Admin preflight** — `POST /api/admin/ai/preflight` (stored under `ai_preflight`, `GET` reads it): ASR `/health` reachability and declared capabilities, real-time factor pooled from the last 10 `session_usage` rows or "not yet measured" (never a synthetic benchmark); one fixed 128-token LLM probe → tokens/s; the declared window and chunk count; the capability gaps; a **throughput class** by band and an **expected processing time for a 3.5 h / 5-speaker session** (prefill charged at 10× generation — measured ratios were 31×/11×/2.8×; validated to within ~30 % on three configurations). `AdminHardwareProfile` panel with a Run button. - **Bands, refined from measurement**: `< 10` / `10–40` / `> 40` tok/s, named by throughput not silicon — a GPU-less 9800X3D measures 15.6 and does not need the long timeouts, so "CPU-class" would have been wrong in both directions; only the bottom band maps to the `cpu` timeouts. Lower bound raised from 8 because one beat extraction on the CPU recipe (~5.7k prompt + ~0.8k completion tokens) crowds the 300 s default at 10 tok/s. - **Measured** (`python -m evals --provider live --record`, four synthetic fixtures, one run each, `history.jsonl` tagged `47c31d7-353-{gpu,cpu}`): | | GPU: RTX 3090, qwen3.5 9B Q8, 131k window | CPU: Ryzen 9800X3D 8 threads, Qwen2.5-3B Q4_K_M, 8k window | |---|---|---| | coverage | **0.850** | **0.500** | | attribution | 0.725 | 0.833 | | chronology | 0.975 | 1.000 | | beat validation | **0.972** | **0.904** | | tok/s in pipeline | 65 | 18 | | wall per 5-min fixture | 30–49 s | 150–186 s | Only coverage degrades on the small model: chronology and attribution come from code-side sort and validator, and beat validation holds because the schema is a decoding grammar. The CPU summary is shorter, not wronger — the issue's thesis, measured. A generic QEMU vCPU without AVX2 (the dev box) measured 0.3 tok/s and is documented as the floor no profile rescues. - **Docs**: OPERATIONS "Hardware profiles" — three recipes (CPU-only, modest GPU, workstation GPU) with model, hardware, timing and scores; the modest-GPU row honestly empty with the hardware and command that fills it; `docs/examples/docker-compose.cpu-llm.yml` as the runnable CPU-only configuration; the commands that produced every number. **Weakness stated:** the private hand-verified session is not present on this machine or the dev box, so both runs used the four synthetic five-minute fixtures — neither exercised windowing, and the coverage figures describe a short transcript. ASR real-time factor is not yet measured anywhere (no `session_usage` rows exist yet). Rebased over #350 and #356 (config and import conflicts, both sides kept). ## Verification - Backend **2064 passed, 10 skipped**; frontend **497 passed**; ruff and eslint clean (one pre-existing warning); version sync OK; the CI evals gate passes. No migration. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot scheduled this pull request to auto merge when all checks succeed

2026-09-05 04:35:18 +00:00

claude-bot referenced this pull request

2026-09-05 04:37:17 +00:00

[Backend] Evaluate ASR and LLM providers against the golden corpus before committing #360

rbrooks force-pushed feat/353-354-self-hosted-profiles from 93caca72e7

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 1m22s

Details

CI / Bot/backend version sync (pull_request) Successful in 23s

Details

CI / Backend lint (ruff) (pull_request) Successful in 24s

Details