[Backend] Publish hardware profiles and expected quality for self-hosted deployments #354

New issue

Closed

opened 2026-08-25 20:39:16 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-08-25 20:39:16 +00:00

Contributor

Found in the August 2026 session lifecycle review (#319).

Why

A self-hoster currently has no way to know what hardware they need or what quality to expect. They discover it by recording a three-hour session and being disappointed — which is the most expensive possible feedback loop.

Proposed fix

Document a small set of named profiles: CPU-only, modest GPU, and a workstation-class GPU. For each, record the model choices, approximate processing time for a 3.5 h / 5-speaker session, the practical context window, and golden-corpus accuracy scores including the beat validation rate.

Add a preflight check to Admin → Bot Settings that measures the configured endpoints and reports which profile the deployment resembles, so the answer arrives before the first real session rather than after it.

Acceptance criteria

At least three named profiles documented with model, hardware, timing, and accuracy
Numbers come from the golden-corpus harness, not estimates
Admin preflight reports the detected profile and any capability gaps
Documented in docs/OPERATIONS.md and referenced from the admin UI

Found in the August 2026 session lifecycle review (#319). ## Why A self-hoster currently has no way to know what hardware they need or what quality to expect. They discover it by recording a three-hour session and being disappointed — which is the most expensive possible feedback loop. ## Proposed fix Document a small set of named profiles: CPU-only, modest GPU, and a workstation-class GPU. For each, record the model choices, approximate processing time for a 3.5 h / 5-speaker session, the practical context window, and golden-corpus accuracy scores including the beat validation rate. Add a preflight check to Admin → Bot Settings that measures the configured endpoints and reports which profile the deployment resembles, so the answer arrives before the first real session rather than after it. ## Acceptance criteria - [ ] At least three named profiles documented with model, hardware, timing, and accuracy - [ ] Numbers come from the golden-corpus harness, not estimates - [ ] Admin preflight reports the detected profile and any capability gaps - [ ] Documented in `docs/OPERATIONS.md` and referenced from the admin UI

claude-bot added this to the v4.2.0 - AI Provider Abstraction & BYO-AI milestone

2026-08-25 20:39:16 +00:00

claude-bot added the

enhancement

backend

labels

2026-08-25 20:39:16 +00:00

claude-bot referenced this issue from a pull request that will close it,

2026-09-05 04:34:46 +00:00

feat: self-hosted first-class — a CPU profile, configurable AI timeouts, an admin hardware preflight, and measured hardware profiles (#353, #354) #500

claude-bot closed this issue

2026-09-05 05:02:28 +00:00

claude-bot referenced this issue from a commit

2026-09-05 05:02:29 +00:00

Merge PR #500: self-hosted first-class — CPU profile, configurable timeouts, hardware preflight, measured profiles (#353, #354)

claude-bot commented

2026-09-05 05:03:08 +00:00

Author

Contributor

Landed in PR #500 (merged 2026-09-05, CI green).

Three named profiles documented in docs/OPERATIONS.md → "Hardware profiles": CPU-only (Ryzen 9800X3D, 3B Q4 — measured), modest GPU (the hardware and command that would fill it; honestly empty), workstation GPU (RTX 3090, qwen3.5 9B Q8 — measured), each with model, hardware, approximate processing time, practical window and golden-corpus scores including beat validation rate.
Numbers come from the harness, not estimates — python -m evals --provider live --record, history.jsonl tags 47c31d7-353-gpu / 47c31d7-353-cpu, the commands reproduced in the docs.
Admin preflight — POST /api/admin/ai/preflight measures the ASR server's reachability and declared capabilities, pools the real-time factor from the last ten session_usage rows (or says "not yet measured" rather than benchmarking synthetically), probes the LLM with one fixed 128-token prompt for tokens/s, and reports the throughput class (< 10 / 10–40 / > 40 tok/s — named by throughput rather than silicon, because a GPU-less 9800X3D measures 15.6 and does not need the long timeouts), the capability gaps, and an expected processing time for a 3.5 h / 5-speaker session (prefill charged at 10× generation; validated to within ~30 % on three configurations). AdminHardwareProfile panel with a Run button; the result is stored and read back.
Referenced from the admin UI to the OPERATIONS section.

Not measured yet: ASR real-time factor (no session_usage rows exist on any deployment yet — the preflight will show it after the first processed session on v4.2.0).

Landed in **PR #500** (merged 2026-09-05, CI green). - **Three named profiles documented** in `docs/OPERATIONS.md` → "Hardware profiles": CPU-only (Ryzen 9800X3D, 3B Q4 — measured), modest GPU (the hardware and command that would fill it; honestly empty), workstation GPU (RTX 3090, qwen3.5 9B Q8 — measured), each with model, hardware, approximate processing time, practical window and golden-corpus scores including beat validation rate. - **Numbers come from the harness, not estimates** — `python -m evals --provider live --record`, `history.jsonl` tags `47c31d7-353-gpu` / `47c31d7-353-cpu`, the commands reproduced in the docs. - **Admin preflight** — `POST /api/admin/ai/preflight` measures the ASR server's reachability and declared capabilities, pools the real-time factor from the last ten `session_usage` rows (or says "not yet measured" rather than benchmarking synthetically), probes the LLM with one fixed 128-token prompt for tokens/s, and reports the throughput class (`< 10` / `10–40` / `> 40` tok/s — named by throughput rather than silicon, because a GPU-less 9800X3D measures 15.6 and does not need the long timeouts), the capability gaps, and an expected processing time for a 3.5 h / 5-speaker session (prefill charged at 10× generation; validated to within ~30 % on three configurations). `AdminHardwareProfile` panel with a Run button; the result is stored and read back. - Referenced from the admin UI to the OPERATIONS section. Not measured yet: ASR real-time factor (no `session_usage` rows exist on any deployment yet — the preflight will show it after the first processed session on v4.2.0).

claude-bot referenced this issue

2026-09-05 06:04:34 +00:00

chore(release): v4.2.0 — AI Provider Abstraction & BYO-AI #504

claude-bot referenced this issue

2026-09-05 07:31:47 +00:00

[Backend] Hardware preflight's LLM probe fails on reasoning models, so the profile reads "Not measured" (#354) #507

claude-bot referenced this issue

2026-09-05 07:31:53 +00:00

[Backend] session_usage.asr_model is left empty while llm_model is populated (#357) #508