feat: self-hosted first-class — a CPU profile, configurable AI timeouts, an admin hardware preflight, and measured hardware profiles (#353, #354) #500

Merged

claude-bot merged 7 commits from feat/353-354-self-hosted-profiles into main 2026-09-05 05:02:28 +00:00

Conversation 0 Commits 7 Files changed 26 +2877 -45

claude-bot commented 2026-09-05 04:34:46 +00:00

Contributor

Copy link

Closes #353. Closes #354.

Self-hosting without a GPU is now a supported, measured configuration rather than a hope, and a self-hoster can find out what their hardware will do before the first three-hour session rather than after it.

What

• Timeouts are configuration (app/config.py, one accessor ai_timeouts()): QB_LLM_TIMEOUT_SECONDS, QB_LLM_PROSE_TIMEOUT_SECONDS, QB_ASR_TIMEOUT_SECONDS, and QB_AI_PROFILE ∈ {auto, gpu, cpu}. cpu multiplies the defaults by 6; an explicitly set value is used as written. Celery task limits derive from the timeouts (task_limits(): on gpu this reproduces #398's 6 h exactly; the invariant is that a task limit can never be shorter than the calls it contains). The admin test button and the monthly canary scale too — both could only ever report a working CPU endpoint as broken. auto follows the last preflight; the Celery limits are read at worker import, so a CPU deployment should pin cpu (stated in code, .env.example, docs and the panel). Two seams fixed on the way: #356's call_lease_ttl was handed base constants at three sites (a lease would have expired mid-call on the cpu profile), and #350's adapter had its own hardcoded 600 s.

• Grammar-constrained beats on llama.cpp, proven on the wire: test_beat_schema_grammar.py drives the real extract_beats through the conformance fake and asserts the closed schema under both spellings, temperature: 0, thinking disabled. Nothing was dropping it.

• Admin preflight — POST /api/admin/ai/preflight (stored under ai_preflight, GET reads it): ASR /health reachability and declared capabilities, real-time factor pooled from the last 10 session_usage rows or "not yet measured" (never a synthetic benchmark); one fixed 128-token LLM probe → tokens/s; the declared window and chunk count; the capability gaps; a throughput class by band and an expected processing time for a 3.5 h / 5-speaker session (prefill charged at 10× generation — measured ratios were 31×/11×/2.8×; validated to within ~30 % on three configurations). AdminHardwareProfile panel with a Run button.

• Bands, refined from measurement: < 10 / 10–40 / > 40 tok/s, named by throughput not silicon — a GPU-less 9800X3D measures 15.6 and does not need the long timeouts, so "CPU-class" would have been wrong in both directions; only the bottom band maps to the cpu timeouts. Lower bound raised from 8 because one beat extraction on the CPU recipe (~5.7k prompt + ~0.8k completion tokens) crowds the 300 s default at 10 tok/s.

• Measured (python -m evals --provider live --record, four synthetic fixtures, one run each, history.jsonl tagged 47c31d7-353-{gpu,cpu}):

  	GPU: RTX 3090, qwen3.5 9B Q8, 131k window	CPU: Ryzen 9800X3D 8 threads, Qwen2.5-3B Q4_K_M, 8k window
  coverage	0.850	0.500
  attribution	0.725	0.833
  chronology	0.975	1.000
  beat validation	0.972	0.904
  tok/s in pipeline	65	18
  wall per 5-min fixture	30–49 s	150–186 s

  Only coverage degrades on the small model: chronology and attribution come from code-side sort and validator, and beat validation holds because the schema is a decoding grammar. The CPU summary is shorter, not wronger — the issue's thesis, measured. A generic QEMU vCPU without AVX2 (the dev box) measured 0.3 tok/s and is documented as the floor no profile rescues.

• Docs: OPERATIONS "Hardware profiles" — three recipes (CPU-only, modest GPU, workstation GPU) with model, hardware, timing and scores; the modest-GPU row honestly empty with the hardware and command that fills it; docs/examples/docker-compose.cpu-llm.yml as the runnable CPU-only configuration; the commands that produced every number.

Weakness stated: the private hand-verified session is not present on this machine or the dev box, so both runs used the four synthetic five-minute fixtures — neither exercised windowing, and the coverage figures describe a short transcript. ASR real-time factor is not yet measured anywhere (no session_usage rows exist yet).

Rebased over #350 and #356 (config and import conflicts, both sides kept).

Verification

• Backend 2064 passed, 10 skipped; frontend 497 passed; ruff and eslint clean (one pre-existing warning); version sync OK; the CI evals gate passes. No migration.

🤖 Generated with Claude Code

Closes #353. Closes #354. Self-hosting without a GPU is now a supported, measured configuration rather than a hope, and a self-hoster can find out what their hardware will do before the first three-hour session rather than after it. ## What - **Timeouts are configuration** (`app/config.py`, one accessor `ai_timeouts()`): `QB_LLM_TIMEOUT_SECONDS`, `QB_LLM_PROSE_TIMEOUT_SECONDS`, `QB_ASR_TIMEOUT_SECONDS`, and **`QB_AI_PROFILE`** ∈ {`auto`, `gpu`, `cpu`}. `cpu` multiplies the defaults by 6; an explicitly set value is used as written. Celery task limits derive from the timeouts (`task_limits()`: on `gpu` this reproduces #398's 6 h exactly; the invariant is that a task limit can never be shorter than the calls it contains). The admin test button and the monthly canary scale too — both could only ever report a working CPU endpoint as broken. `auto` follows the last preflight; the Celery limits are read at worker import, so a CPU deployment should pin `cpu` (stated in code, `.env.example`, docs and the panel). Two seams fixed on the way: #356's `call_lease_ttl` was handed base constants at three sites (a lease would have expired mid-call on the cpu profile), and #350's adapter had its own hardcoded 600 s. - **Grammar-constrained beats on llama.cpp, proven on the wire**: `test_beat_schema_grammar.py` drives the real `extract_beats` through the conformance fake and asserts the closed schema under both spellings, `temperature: 0`, thinking disabled. Nothing was dropping it. - **Admin preflight** — `POST /api/admin/ai/preflight` (stored under `ai_preflight`, `GET` reads it): ASR `/health` reachability and declared capabilities, real-time factor pooled from the last 10 `session_usage` rows or "not yet measured" (never a synthetic benchmark); one fixed 128-token LLM probe → tokens/s; the declared window and chunk count; the capability gaps; a **throughput class** by band and an **expected processing time for a 3.5 h / 5-speaker session** (prefill charged at 10× generation — measured ratios were 31×/11×/2.8×; validated to within ~30 % on three configurations). `AdminHardwareProfile` panel with a Run button. - **Bands, refined from measurement**: `< 10` / `10–40` / `> 40` tok/s, named by throughput not silicon — a GPU-less 9800X3D measures 15.6 and does not need the long timeouts, so "CPU-class" would have been wrong in both directions; only the bottom band maps to the `cpu` timeouts. Lower bound raised from 8 because one beat extraction on the CPU recipe (~5.7k prompt + ~0.8k completion tokens) crowds the 300 s default at 10 tok/s. - **Measured** (`python -m evals --provider live --record`, four synthetic fixtures, one run each, `history.jsonl` tagged `47c31d7-353-{gpu,cpu}`): | | GPU: RTX 3090, qwen3.5 9B Q8, 131k window | CPU: Ryzen 9800X3D 8 threads, Qwen2.5-3B Q4_K_M, 8k window | |---|---|---| | coverage | **0.850** | **0.500** | | attribution | 0.725 | 0.833 | | chronology | 0.975 | 1.000 | | beat validation | **0.972** | **0.904** | | tok/s in pipeline | 65 | 18 | | wall per 5-min fixture | 30–49 s | 150–186 s | Only coverage degrades on the small model: chronology and attribution come from code-side sort and validator, and beat validation holds because the schema is a decoding grammar. The CPU summary is shorter, not wronger — the issue's thesis, measured. A generic QEMU vCPU without AVX2 (the dev box) measured 0.3 tok/s and is documented as the floor no profile rescues. - **Docs**: OPERATIONS "Hardware profiles" — three recipes (CPU-only, modest GPU, workstation GPU) with model, hardware, timing and scores; the modest-GPU row honestly empty with the hardware and command that fills it; `docs/examples/docker-compose.cpu-llm.yml` as the runnable CPU-only configuration; the commands that produced every number. **Weakness stated:** the private hand-verified session is not present on this machine or the dev box, so both runs used the four synthetic five-minute fixtures — neither exercised windowing, and the coverage figures describe a short transcript. ASR real-time factor is not yet measured anywhere (no `session_usage` rows exist yet). Rebased over #350 and #356 (config and import conflicts, both sides kept). ## Verification - Backend **2064 passed, 10 skipped**; frontend **497 passed**; ruff and eslint clean (one pre-existing warning); version sync OK; the CI evals gate passes. No migration. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 6 commits 2026-09-05 04:34:46 +00:00

feat(config): AI timeouts and the cpu profile ... 2be1f978e6

Every deadline in the AI pipeline was a module constant chosen against a
GPU: five minutes for a structured call, ten for a transcription track,
sixty seconds for the monthly canary, thirty for the admin "Test
connection" button. On CPU-only hardware those are not slow budgets, they
are impossible ones — so a self-hoster without a GPU saw *failures*
rather than patience, with a recording already made and nothing naming
the cause. Self-hosting has to stay first-class, and that means the CPU
path has to be correct, not merely runnable.

QB_AI_PROFILE (auto | gpu | cpu) is now the setting, and the timeouts
follow from it; cpu multiplies the three defaults by 6. The three are
also individually settable — QB_LLM_TIMEOUT_SECONDS,
QB_LLM_PROSE_TIMEOUT_SECONDS, QB_ASR_TIMEOUT_SECONDS — and an explicit
value is used exactly as written, because a number the operator measured
and wrote down is a decision, not a default to scale.

All of it resolves through one function, ai_timeouts(), rather than
through five constants in three files. Those five had already drifted:
the *prose* transports, which carry the longest calls in the product,
were running on _LLM_TIMEOUT (5 min) while the constant actually named
_PROSE_TIMEOUT (10 min) was reached from a single call site. Nothing
could notice, because nothing ever read them side by side.

The Celery limits are derived from the timeouts rather than written
beside them. The invariant is a relationship, not two numbers: a task
limit shorter than the provider calls the task is allowed to make kills
a run that was working exactly as intended. On the gpu profile the
derivation reproduces #398's 6 h / 6 h 10 m exactly, so nothing moves for
an existing deployment.

auto resolves to gpu unless a preflight has measured this deployment as
CPU-class. That resolution is process-local by necessity — the transports
have no database session and celery_app has none at import — so the
consequence is written down rather than hidden: auto moves the
per-request timeouts, and a deployment that wants the task limits moved
too pins QB_AI_PROFILE=cpu.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(admin): hardware preflight ... c1cd0c3619

A self-hoster had no way to learn what hardware they needed or what
quality to expect except by recording a three-hour session and being
disappointed. Everything in this area declares — the provider adapters
declare capabilities, #358 turns those into gaps, #359 proves the
endpoint still answers — and none of it measures speed, which is the
whole question someone asks before they buy anything.

POST /api/admin/ai/preflight measures and stores; GET reads back the
last result. Deliberately the smallest thing that yields a number: one
/health round trip, one 128-token generation on a fixed prompt through
generate_structured_text, so what is timed is the path a real session
takes rather than a simplified stand-in.

ASR is not benchmarked here at all. A synthetic clip's real-time factor
is per-request overhead, not throughput, so the RTF comes from
session_usage (#357) — the operator's own recordings — pooled over the
last ten runs rather than averaged as ratios, because a five-minute test
recording and a four-hour session are not equally informative. A
deployment with no runs reads "not yet measured". An invented number
would be worse than an absence.

The bands are named for throughput rather than silicon: CPU-class under
10 tok/s, modest GPU or fast CPU 10-40, workstation above 40. The
measurements are why. A Ryzen 7 9800X3D with no GPU runs this pipeline
at 15.6 tok/s on the probe and does not need the longer timeouts;
calling that "CPU-class" would be wrong in both directions. The lower
boundary is 10 rather than the 8 first proposed because a beat
extraction on the CPU tier's own recommended shape — 3B, 8k window —
crowds the 300-second timeout below that.

Only the bottom band asks for the cpu timeouts. unknown maps to gpu: an
endpoint that could not answer a 128-token request has a configuration
problem, and quietly sextupling every deadline would delay the moment
anyone finds out.

Nothing here 4xxs or 500s on a provider that is missing or unreachable.
That is the finding, and an admin who opened this page to learn what is
wrong should be told rather than shown an error.

Startup reloads the stored verdict, because auto is only worth having if
it survives a deploy.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(frontend): show what this deployment's hardware can actually do ... 1546e06889

Admin -> Bot Settings already says what the providers can express (#358)
and what they have cost (#357). Neither answers "how long will tonight's
recording take", which is the question a self-hoster has before they buy
a GPU or decide they do not need one.

A Run preflight button rather than a measurement on mount: it costs a
real generation against a possibly-metered provider and it is slow on
exactly the hardware it exists to describe. The result carries the date
it was taken and the panel says "as measured on ...", because a stale
number that knows it is stale beats a fresh one nobody waited for.

Absence is rendered rather than hidden. A deployment with no processed
sessions has no real-time factor, and the slot says "not yet measured"
where the number would go — the same choice the backend makes, for the
same reason: an empty slot invites the operator to process a session, a
fabricated 0.0 tells them transcription is instant.

The one case that must not be silent: a CPU-class measurement while the
process is still on GPU timings. The per-request timeouts follow the
measurement but the Celery limits are read at worker start, so the panel
says to pin QB_AI_PROFILE=cpu and restart.

Mounted directly under the capability panel — what your configuration
can do, then how fast it does it, then what that has cost.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

test: the profiles, the derivation, the grammar and the preflight ... 1e3f92d5c2

Three things had to be pinned.

**The timeouts.** That cpu scales defaults and never an explicit value;
that the Celery limits are derived and, on the gpu profile, are still
byte-identical to the 6 h / 6 h 10 m #398 chose — if that moves, an
existing deployment's behaviour moved; that a shortened timeout cannot
shrink the pathology bound; that auto reads the stored preflight and
falls back to gpu. The observed profile is process state, so an autouse
fixture clears it: leaking one would turn a failing test into three.

**The grammar.** This is the property that makes a CPU tier honest
rather than merely runnable. A 3B model will find fewer events than a 9B
— what must not happen is that it is wrong invisibly. llama.cpp compiles
the json_schema into a GBNF grammar, so a conforming server cannot emit
a malformed or unexpectedly-shaped beat; chronology is a code-side sort
and attribution a code-side validator; so a weak model loses beats to
*validation* rather than to plausible invention. The test asserts the
beat-extraction schema really reaches the wire in both spellings, driven
by the conformance suite's own MockTransport fake so what is checked is
the body httpx encoded, not the kwargs a caller passed. It has silently
not been true before: #281 found llama.cpp's documented flat spelling is
accepted and ignored, so every json_schema call in production was doing
nothing while every unit test stayed green.

**The preflight.** The bands, including the three configurations
actually measured for the docs and both boundaries; that unknown does
not relax the timeouts; that the extrapolation charges prompt and
completion at different rates and accounts for what VAD removed; that a
404 /health is reachable and a refused connection is a finding rather
than an exception; that a provider reporting no tokens leaves the rate
unmeasured rather than guessed; that a CPU-class measurement reaches
auto. HTTP is faked at the AsyncClient constructor throughout.

Admin.test.jsx gains the two new api mocks and the empty-preflight
default, so Bot Settings tests do not fail inside an unrelated panel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

chore(evals): record the profile measurements ... 30bcf5c2c2

The harness scored accuracy and never recorded what the accuracy cost.
That cannot answer the question the hardware profiles exist to settle,
which is not "is this model accurate" but "is it accurate enough for
what it costs me in wall time" — two configurations that both score 0.8
are not the same deployment if one takes ninety seconds and the other
takes fifty minutes.

Each history record now carries wall_seconds, llm_calls, prompt_tokens,
completion_tokens and tokens_per_second. Tokens come from the same
usage_meter scope production uses (#357) rather than a second counter, so
they mean on the bench what they mean on a real run. They are
deliberately absent from render_report, which is documented as
deterministic — two stub runs over the same fixtures must print
identical text — and variation is the whole point of the history file.

Eight recorded runs: the four synthetic fixtures against the RTX 3090
router (qwen3.5-9B Q8) and against llama.cpp on a CPU with no GPU at all
(Qwen2.5-3B-Instruct Q4_K_M, 8k window). They are what docs/OPERATIONS.md
quotes, and they say something worth having measured:

    coverage    0.85 -> 0.50
    attribution 0.73 -> 0.83
    chronology  0.98 -> 1.00
    beat valid. 0.97 -> 0.90
    per fixture 38 s -> 168 s, 65.4 -> 18.0 tok/s

Only coverage falls. Chronology holds because it is a code-side sort,
attribution holds because it is a code-side validator, and the beats
that survive validation are still nearly all valid because the schema is
a decoding grammar. The small model's summary is shorter, not wronger —
which is the property that makes a CPU tier honest, now measured rather
than asserted.

The CPU run also made five LLM calls per fixture where the GPU made
three: the dropped-event recovery pass firing, visible in llm_calls.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs(operations): hardware profiles, measured rather than asserted ... 93caca72e7

OPERATIONS.md could not answer the first question a self-hoster has —
what hardware do I need, and what will I get — except by inviting them to
record a three-hour session and find out. Three named profiles now
answer it, with the harness command that produced every number.

The headline is not what anyone expects: a modern desktop CPU runs this
pipeline fine. A Ryzen 7 9800X3D with no GPU summarises at ~18 tok/s and
scores better on attribution and chronology than the RTX 3090 does. What
it loses is coverage — 0.85 to 0.50 — which is a trade an operator can
look at and decide about rather than a surprise in a summary.

Why that shape holds is the part worth writing down: chronology is a
code-side sort, attribution a code-side validator, and the JSON schema is
a decoding grammar on llama.cpp, so a weaker model loses beats to
validation rather than producing a confident invention. Shorter, not
wronger.

The modest-GPU row is honestly empty. No 8-16 GB card was available;
rather than guess a row into a table of measured ones, it says what
hardware would settle it and what command to run.

A floor is documented too, because it is the one configuration measured
that genuinely does not work: six generic QEMU vCPUs exposing nothing
above SSE4.2 run the same model at 2.2 tok/s prompt and 0.8 tok/s
generation — roughly 100x down on the half a summarisation prompt is
made of. A 3.5-hour session would take ten hours there and a single
extraction call would exceed even the cpu profile's timeout. No setting
rescues that; the fix is the hypervisor's CPU model. "grep avx2
/proc/cpuinfo before you buy" is a more useful sentence than any
timeout.

docs/examples/docker-compose.cpu-llm.yml makes the CPU tier runnable
rather than merely described: one llama.cpp service on the CPU, joined to
the stack network, named `llamacpp` so it lands on the SSRF allowlist
without loosening anything, with the window and its Bot Settings twin
spelled out — over-declaring that is the dangerous direction, because
llama.cpp answers an over-long prompt by discarding its front.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed 2026-09-05 04:35:18 +00:00

claude-bot referenced this pull request 2026-09-05 04:37:17 +00:00

[Backend] Evaluate ASR and LLM providers against the golden corpus before committing #360

rbrooks force-pushed feat/353-354-self-hosted-profiles from 93caca72e7 to 0981e3fe57 2026-09-05 04:54:07 +00:00 Compare

claude-bot merged commit 8c85602a78 into main 2026-09-05 05:02:28 +00:00

claude-bot deleted branch feat/353-354-self-hosted-profiles 2026-09-05 05:02:28 +00:00

claude-bot referenced this pull request from a commit 2026-09-05 05:02:29 +00:00

Merge PR #500: self-hosted first-class — CPU profile, configurable timeouts, hardware preflight, measured profiles (#353, #354)

claude-bot referenced this pull request 2026-09-05 05:03:00 +00:00

[Backend] Keep self-hosted LLM first-class, including a CPU-only path #353

claude-bot referenced this pull request 2026-09-05 05:03:08 +00:00

[Backend] Publish hardware profiles and expected quality for self-hosted deployments #354

claude-bot referenced this pull request 2026-09-05 06:04:34 +00:00

chore(release): v4.2.0 — AI Provider Abstraction & BYO-AI #504

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backend

Issue related to the backend

  

bot

Issue related to the bot

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

frontend

Issue related to the frontend

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

recording

Issue related to the recording system

  

security

Security hardening / vulnerability work

  

wontfix

This won't be fixed

No labels

backend

bot

bug

duplicate

enhancement

frontend

help wanted

invalid

question

recording

security

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/Quest-Board!500

No description provided.