[Backend] Keep self-hosted LLM first-class, including a CPU-only path #353

New issue

Closed

opened 2026-08-25 20:39:16 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-08-25 20:39:16 +00:00

Contributor

Found in the August 2026 session lifecycle review (#319).

Why

Self-hosting must remain viable without a GPU. That is a real constraint, not a courtesy: a CPU-only deployment has a much smaller practical context window and much slower generation, and the pipeline has to be correct there, not merely runnable.

This is the strongest argument for the architecture chosen in v4.0.0. Because chronology is guaranteed by a code-side sorted() and attribution is checked by a code-side validator, the accuracy floor does not depend on model size. A small CPU-hosted model loses more beats to validation; it does not silently produce a confidently wrong summary. That property is what makes a CPU tier honest.

Proposed fix

Make the CPU path a supported, tested configuration: llama.cpp on CPU with a small model, GBNF grammar constraint for the beat schema, window sized from the declared context, and generous timeouts throughout the Celery chain so a slow local run does not read as a failure.

Measure it against the golden corpus and publish the numbers — including the beat validation rate, which is the honest quality signal — so a self-hoster can make an informed choice rather than discovering the trade-off in production.

Acceptance criteria

A documented CPU-only configuration runs the full pipeline end to end
Grammar-constrained beat output works on the CPU path
Timeouts across the Celery chain accommodate slow local generation
Golden-corpus scores published for at least one CPU and one GPU local configuration
Degradations are surfaced to the user rather than hidden

Found in the August 2026 session lifecycle review (#319). ## Why Self-hosting must remain viable without a GPU. That is a real constraint, not a courtesy: a CPU-only deployment has a much smaller practical context window and much slower generation, and the pipeline has to be *correct* there, not merely *runnable*. This is the strongest argument for the architecture chosen in v4.0.0. Because chronology is guaranteed by a code-side `sorted()` and attribution is checked by a code-side validator, the accuracy floor does not depend on model size. A small CPU-hosted model loses more beats to validation; it does not silently produce a confidently wrong summary. That property is what makes a CPU tier honest. ## Proposed fix Make the CPU path a supported, tested configuration: llama.cpp on CPU with a small model, GBNF grammar constraint for the beat schema, window sized from the declared context, and generous timeouts throughout the Celery chain so a slow local run does not read as a failure. Measure it against the golden corpus and publish the numbers — including the beat validation rate, which is the honest quality signal — so a self-hoster can make an informed choice rather than discovering the trade-off in production. ## Acceptance criteria - [ ] A documented CPU-only configuration runs the full pipeline end to end - [ ] Grammar-constrained beat output works on the CPU path - [ ] Timeouts across the Celery chain accommodate slow local generation - [ ] Golden-corpus scores published for at least one CPU and one GPU local configuration - [ ] Degradations are surfaced to the user rather than hidden

claude-bot added this to the v4.2.0 - AI Provider Abstraction & BYO-AI milestone

2026-08-25 20:39:16 +00:00

claude-bot added the

enhancement

backend

labels

2026-08-25 20:39:16 +00:00

claude-bot referenced this issue from a pull request that will close it,

2026-09-05 04:34:46 +00:00

feat: self-hosted first-class — a CPU profile, configurable AI timeouts, an admin hardware preflight, and measured hardware profiles (#353, #354) #500

rbrooks referenced this issue from a commit

2026-09-05 04:54:07 +00:00

style: ruff format the adapter after the rebase over #352 (#353)

claude-bot closed this issue

2026-09-05 05:02:28 +00:00

claude-bot referenced this issue from a commit

2026-09-05 05:02:29 +00:00

Merge PR #500: self-hosted first-class — CPU profile, configurable timeouts, hardware preflight, measured profiles (#353, #354)

claude-bot commented

2026-09-05 05:03:00 +00:00

Author

Contributor

Landed in PR #500 (merged 2026-09-05, CI green; 2146 backend / 497 frontend tests after the final rebase).

A documented CPU-only configuration runs the full pipeline end to end — docs/examples/docker-compose.cpu-llm.yml (llama.cpp server, Qwen2.5-3B-Instruct Q4_K_M, 8k window) plus the Bot Settings values, and the "Hardware profiles" section in OPERATIONS with the exact commands.
Grammar-constrained beat output on the CPU path — proven on the wire by test_beat_schema_grammar.py: the real extract_beats sends the closed schema under both llama.cpp spellings with temperature: 0 and thinking disabled; nothing was dropping it.
Timeouts across the Celery chain — QB_LLM_TIMEOUT_SECONDS, QB_LLM_PROSE_TIMEOUT_SECONDS, QB_ASR_TIMEOUT_SECONDS and QB_AI_PROFILE=cpu (×6 on the defaults), with the Celery task limits derived from them so a task limit can never be shorter than the calls it contains; the admin test button and the monthly canary scale too. auto follows the last preflight, but Celery limits are read at worker import, so a CPU deployment should pin cpu — stated in code, .env.example, docs and the panel.
Golden-corpus scores published for a GPU config (RTX 3090, qwen3.5 9B Q8: coverage 0.850, attribution 0.725, chronology 0.975, beat validation 0.972, 65 tok/s) and a CPU config (Ryzen 9800X3D 8 threads, 3B Q4: 0.500 / 0.833 / 1.000 / 0.904, 18 tok/s). Only coverage degrades on the small model; chronology, attribution and grammar-validated beats hold — the CPU summary is shorter, not wronger, which is this issue's thesis measured. A generic QEMU vCPU without AVX2 measured 0.3 tok/s and is documented as the floor no profile rescues.
Degradations surfaced via #358's panel and the new hardware-profile panel.

Stated weakness: the private hand-verified session is not present on this machine or the dev box, so both runs used the four synthetic five-minute fixtures — neither exercised windowing. Re-running against a real session is the next sharpening step.

Landed in **PR #500** (merged 2026-09-05, CI green; 2146 backend / 497 frontend tests after the final rebase). - **A documented CPU-only configuration runs the full pipeline end to end** — `docs/examples/docker-compose.cpu-llm.yml` (llama.cpp server, Qwen2.5-3B-Instruct Q4_K_M, 8k window) plus the Bot Settings values, and the "Hardware profiles" section in OPERATIONS with the exact commands. - **Grammar-constrained beat output on the CPU path** — proven on the wire by `test_beat_schema_grammar.py`: the real `extract_beats` sends the closed schema under both llama.cpp spellings with `temperature: 0` and thinking disabled; nothing was dropping it. - **Timeouts across the Celery chain** — `QB_LLM_TIMEOUT_SECONDS`, `QB_LLM_PROSE_TIMEOUT_SECONDS`, `QB_ASR_TIMEOUT_SECONDS` and `QB_AI_PROFILE=cpu` (×6 on the defaults), with the Celery task limits derived from them so a task limit can never be shorter than the calls it contains; the admin test button and the monthly canary scale too. `auto` follows the last preflight, but Celery limits are read at worker import, so a CPU deployment should pin `cpu` — stated in code, `.env.example`, docs and the panel. - **Golden-corpus scores published** for a GPU config (RTX 3090, qwen3.5 9B Q8: coverage 0.850, attribution 0.725, chronology 0.975, beat validation 0.972, 65 tok/s) and a CPU config (Ryzen 9800X3D 8 threads, 3B Q4: 0.500 / 0.833 / 1.000 / 0.904, 18 tok/s). Only coverage degrades on the small model; chronology, attribution and grammar-validated beats hold — the CPU summary is shorter, not wronger, which is this issue's thesis measured. A generic QEMU vCPU without AVX2 measured 0.3 tok/s and is documented as the floor no profile rescues. - **Degradations surfaced** via #358's panel and the new hardware-profile panel. Stated weakness: the private hand-verified session is not present on this machine or the dev box, so both runs used the four synthetic five-minute fixtures — neither exercised windowing. Re-running against a real session is the next sharpening step.

claude-bot referenced this issue

2026-09-05 06:04:34 +00:00

chore(release): v4.2.0 — AI Provider Abstraction & BYO-AI #504