fix(backend): probe the LLM with a budget a reasoning model can answer in #509

Merged

claude-bot merged 4 commits from fix/507-preflight-reasoning-probe into main

2026-09-05 15:41:47 +00:00

claude-bot commented

2026-09-05 15:23:13 +00:00

Contributor

Closes #507.

The hardware preflight's LLM probe asked for at most 128 output tokens, which on a reasoning model is no answer at all: the hidden thinking spends the whole budget before the first visible word, llama.cpp returns finish_reason='length' with empty content, and the panel reported "Not measured" on the default self-hosted stack (qwen3.5), beside an endpoint that summarises whole sessions fine.

The output cap is now a capability question. A provider that declares reasoning is probed with PROBE_MAX_TOKENS_REASONING = 2048; everyone else keeps 128. Resolved through llm_provider_for on the config, never sniffed from the URL. Bounded rather than max_tokens=None, so the probe stays bounded in time on exactly the slow hardware it exists to describe.
A provider that declared no reasoning and ran out of budget anyway is retried exactly once with the larger budget, and the result records retried_with_reasoning_budget so the panel can say why the measurement took longer. One retry, never a loop.
Detection is structural: llm_service.OutputBudgetExhausted (a RuntimeError subclass, so every existing caller and test is unchanged) types the llama.cpp raise, and UsageMeter now keeps the per-call truncation flag Usage always carried, which is how the transports that return an empty string instead of raising are recognised. Nothing matches on operator-facing prose.
tokens_per_second still counts reasoning tokens: they decode at the same rate and a real run pays for them through the same meter.
One conditional line on the Admin hardware panel, and OPERATIONS no longer promises a flat 128-token generation.

Nine new tests, asserting on the max_tokens that goes over the wire through the existing MockTransport seam rather than on a patched generate_structured_text. Full backend suite 2321 passed / 13 skipped; frontend 505 passed; ruff and eslint clean. No migration, no contract change.

🤖 Generated with Claude Code

Closes #507. The hardware preflight's LLM probe asked for at most 128 output tokens, which on a reasoning model is no answer at all: the hidden thinking spends the whole budget before the first visible word, llama.cpp returns `finish_reason='length'` with empty content, and the panel reported "Not measured" on the default self-hosted stack (qwen3.5), beside an endpoint that summarises whole sessions fine. - The output cap is now a capability question. A provider that declares `reasoning` is probed with `PROBE_MAX_TOKENS_REASONING = 2048`; everyone else keeps 128. Resolved through `llm_provider_for` on the config, never sniffed from the URL. Bounded rather than `max_tokens=None`, so the probe stays bounded in time on exactly the slow hardware it exists to describe. - A provider that declared no reasoning and ran out of budget anyway is retried exactly once with the larger budget, and the result records `retried_with_reasoning_budget` so the panel can say why the measurement took longer. One retry, never a loop. - Detection is structural: `llm_service.OutputBudgetExhausted` (a `RuntimeError` subclass, so every existing caller and test is unchanged) types the llama.cpp raise, and `UsageMeter` now keeps the per-call truncation flag `Usage` always carried, which is how the transports that return an empty string instead of raising are recognised. Nothing matches on operator-facing prose. - `tokens_per_second` still counts reasoning tokens: they decode at the same rate and a real run pays for them through the same meter. - One conditional line on the Admin hardware panel, and OPERATIONS no longer promises a flat 128-token generation. Nine new tests, asserting on the `max_tokens` that goes over the wire through the existing `MockTransport` seam rather than on a patched `generate_structured_text`. Full backend suite 2321 passed / 13 skipped; frontend 505 passed; ruff and eslint clean. No migration, no contract change. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 4 commits

2026-09-05 15:23:13 +00:00

fix(backend): give the output-cap failure a type, not just a sentence (#507 ) c5f6488fb4

A response that stops at max_tokens before emitting any visible text is a
condition a caller can act on — raise the budget and ask again — but the only
way to recognise it was to match the English message llama.cpp's transport
raises, which exists to be read by an operator and is free to be reworded.

OutputBudgetExhausted subclasses RuntimeError, so every existing `except
RuntimeError` and every test matching on the wording is unchanged; what it adds
is a type, plus the provider and the cap that was in force.

The other three transports do not raise on it at all — they hand back an empty
string — so UsageMeter now keeps the truncation flag Usage has always carried
per call. One metered call plus that flag is enough to tell "answered with
nothing because the budget ran out" from "answered with nothing", without
reading an error message either.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

fix(backend): probe the LLM with a budget a reasoning model can answer in (#507 ) 79f45c32a0

The hardware preflight asked for at most 128 tokens of prose. That is not a
short answer for a thinking model, it is no answer at all: the hidden reasoning
spends the whole budget before the first visible word, the endpoint returns
finish_reason='length' with empty content, and the probe reports a failure. On
the bundled llama.cpp stack serving qwen3.5 — the default self-hosted
configuration — that happened every time, so Admin -> AI showed profile
"unknown", "Not measured", and no expected session time beside an endpoint that
summarises whole sessions perfectly well.

The cap is a capability question rather than a constant, so it is resolved the
way every generation path resolves one, through llm_provider_for on the config:
a provider that declares reasoning gets PROBE_MAX_TOKENS_REASONING (2048),
everything else keeps the cheap 128. Bounded rather than max_tokens=None,
because an uncapped generation is unbounded in time on exactly the slow hardware
this module exists to describe; 2048 still fits inside the deadline at the
slowest rate the bands call viable.

The declaration is per adapter and the model is the operator's choice, so a
provider that declared no reasoning and ran out of budget anyway — the OpenAI
adapter pointed at an o-series model, Ollama serving qwen3 — is retried exactly
once with the larger budget, and the result records
retried_with_reasoning_budget so the panel can explain the longer wait. One
retry, never a loop.

tokens_per_second is still completion tokens over wall clock, which on a
thinking model includes the reasoning tokens: they decode at the same rate and a
real summarisation run pays for them too.

probe_llm still never raises and the result shape only gained keys.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(frontend): say when the preflight needed a second, larger attempt (#507 ) 3f4d338d74

A probe that was retried with a reasoning budget took roughly twice as long and
measured a model that thinks before it writes. Both are worth saying: without
it the panel reports an unexplained wait and a rate that reads low for the
hardware, with nothing on the page connecting the two.

One line under the figures when the backend records
retried_with_reasoning_budget, and a clause in the "measuring" copy so the
longer wait is expected rather than discovered. No redesign.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs: the preflight's output cap is a capability question (#507 )

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 56s

Details

CI / Docker image build (pull_request) Successful in 36s

Details

CI / Bot/backend version sync (pull_request) Successful in 38s

Details

CI / Backend lint (ruff) (pull_request) Successful in 39s

Details

CI / Bot tests and audit (pull_request) Successful in 1m38s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 1m48s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 7m1s

Details

CI / Synthetic session harness (no GPU, no LLM) (pull_request) Successful in 17m32s

Details

cbe72d30af

CHANGELOG entry under Unreleased, and OPERATIONS' "Measuring your own
deployment" no longer promises a flat 128-token generation — it names both
budgets, says which model gets which, and records that the reported rate counts
reasoning tokens on purpose.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed

2026-09-05 15:23:24 +00:00

claude-bot merged commit f33c2d9aa1 into main

2026-09-05 15:41:47 +00:00

claude-bot deleted branch fix/507-preflight-reasoning-probe

2026-09-05 15:41:47 +00:00

claude-bot referenced this pull request from a commit

2026-09-05 15:41:48 +00:00

Merge PR #509: fix(backend): probe the LLM with a budget a reasoning model can answer in

claude-bot referenced this pull request

2026-09-05 15:44:27 +00:00

[Backend] Hardware preflight's LLM probe fails on reasoning models, so the profile reads "Not measured" (#354) #507