fix(backend): rate the LLM on what it decoded, not on whether it finished #512

Merged

claude-bot merged 4 commits from fix/507-rate-from-exhausted-budget into main

2026-09-05 17:05:11 +00:00

claude-bot commented

2026-09-05 16:48:54 +00:00

Contributor

Closes #507 (second attempt; the v4.2.1 fix in PR #509 did not hold on the bundled stack).

Live on dev after v4.2.1, the probe gave qwen3.5 the 2048-token reasoning budget and the model spent all of it thinking: 60.9 s, finish_reason='length', no visible text, "Not measured" again. Raising the cap is the wrong lever, since the thinking is unbounded on this prompt and any cap a 3090 can finish inside is a deadline timeout on a CPU box.

The premise was wrong rather than the number. 2048 tokens in 60.9 s is 33.6 tokens per second, the only figure the preflight exists to obtain. Reasoning tokens decode at the same rate as visible ones, and measure_llm_tokens_per_run already counts them through the same meter. The probe was discarding a good throughput sample because the model had not finished its sentence.

The llama.cpp transport now records usage on the meter before raising OutputBudgetExhausted. It was the only transport raising ahead of _check_usage, which is why the probe saw completion_tokens: null while the transport's own log printed 2048. Recorded directly rather than via _check_usage, which under json_mode raises before reaching the meter and would replace the typed exception. Session cost telemetry gains these tokens too.
An exhausted budget with reported tokens is a successful measurement: ok: true, visible_text: false, rate filled in, error: "", plus a note saying where the sample came from. An exhausted budget with no usage at all stays ok: false with the error, so a broken endpoint still reads as broken.
PROBE_MAX_TOKENS_REASONING 2048 → 512: a rate sample, not room for an answer (about 100 s at 5 tok/s against the 300 s deadline). PROBE_MAX_TOKENS stays 128; the single retry for undeclared reasoning providers stays and never replaces a measurement with a failure.
run_preflight already read the rate rather than probe["ok"], so the band, expected-session estimate and timeout profile follow; now stated in code and covered end to end.
Admin → Hardware profile explains a rate measured without a finished answer. OPERATIONS and the changelog match.

Backend 2335 passed / 13 skipped, frontend 509 passed, ruff and eslint clean. No migration, no contract change. Goes out as v4.2.2; verification is the dev preflight showing a band at roughly 33 tok/s with the note.

🤖 Generated with Claude Code

Closes #507 (second attempt; the v4.2.1 fix in PR #509 did not hold on the bundled stack). Live on dev after v4.2.1, the probe gave qwen3.5 the 2048-token reasoning budget and the model spent all of it thinking: 60.9 s, `finish_reason='length'`, no visible text, "Not measured" again. Raising the cap is the wrong lever, since the thinking is unbounded on this prompt and any cap a 3090 can finish inside is a deadline timeout on a CPU box. The premise was wrong rather than the number. 2048 tokens in 60.9 s **is** 33.6 tokens per second, the only figure the preflight exists to obtain. Reasoning tokens decode at the same rate as visible ones, and `measure_llm_tokens_per_run` already counts them through the same meter. The probe was discarding a good throughput sample because the model had not finished its sentence. - The llama.cpp transport now records usage on the meter **before** raising `OutputBudgetExhausted`. It was the only transport raising ahead of `_check_usage`, which is why the probe saw `completion_tokens: null` while the transport's own log printed 2048. Recorded directly rather than via `_check_usage`, which under `json_mode` raises before reaching the meter and would replace the typed exception. Session cost telemetry gains these tokens too. - An exhausted budget with reported tokens is a successful measurement: `ok: true`, `visible_text: false`, rate filled in, `error: ""`, plus a `note` saying where the sample came from. An exhausted budget with no usage at all stays `ok: false` with the error, so a broken endpoint still reads as broken. - `PROBE_MAX_TOKENS_REASONING` 2048 → 512: a rate sample, not room for an answer (about 100 s at 5 tok/s against the 300 s deadline). `PROBE_MAX_TOKENS` stays 128; the single retry for undeclared reasoning providers stays and never replaces a measurement with a failure. - `run_preflight` already read the rate rather than `probe["ok"]`, so the band, expected-session estimate and timeout profile follow; now stated in code and covered end to end. - Admin → Hardware profile explains a rate measured without a finished answer. OPERATIONS and the changelog match. Backend 2335 passed / 13 skipped, frontend 509 passed, ruff and eslint clean. No migration, no contract change. Goes out as v4.2.2; verification is the dev preflight showing a band at roughly 33 tok/s with the note. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 4 commits

2026-09-05 16:48:54 +00:00

fix(backend): keep the tokens when the output budget runs out (#507 ) 0889b66514

llama.cpp is the only transport that raises on empty content at
finish_reason='length', and it raised before `_check_usage` ever ran — so
the one path where a reasoning model reports a full budget's worth of
decoding was also the one path that threw the usage away. The other three
reach `_check_usage` on the same event and record it.

The answer is missing; the tokens are not. The model decoded them at this
endpoint's real speed and occupied the slot for as long as it took, which
is exactly the sample the hardware preflight's generation rate is made of
(2048 tokens in 60.9 s on the bundled stack is 33.6 tok/s), and it is also
the GPU time per-session cost telemetry has been silently under-counting.

Recorded straight onto the meter rather than through `_check_usage`:
under json_mode that helper raises on a truncated response before it
reaches the meter, and its checks are about whether returned content can
be trusted. There is no content here to trust, only a measurement to keep.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

fix(backend): rate the LLM on what it decoded, not on whether it finished (#507 ) 140d4bd368

v4.2.1 gave a declared reasoning provider a 2048-token probe budget on the
theory that a thinking model would finish inside one. Live on the bundled
llama.cpp stack serving qwen3.5 — the default self-hosted configuration —
it did not: the model thought for all 2048 tokens, 60.9 seconds of it, and
the panel reported "Not measured" a second time.

Raising the cap again is the wrong lever twice over. On this prompt the
thinking is unbounded, so no cap is reliably "enough"; and any budget large
enough for a 3090 to finish inside is a deadline timeout on a CPU box,
where 2048 tokens at 5 tok/s is 400 seconds against a 300-second limit.

So the probe stopped requiring an answer. 2048 tokens in 60.9 seconds *is*
33.6 tokens per second, which is the only number this module exists to
obtain — reasoning tokens are decoded at the same rate as visible ones and
a real summarisation run pays for them through the same meter. An
exhausted budget that reports completion tokens now comes back `ok` with
the rate filled in, `visible_text: false` and a note saying where the
sample came from; only a generation that reports no tokens at all is still
a failure, so a broken endpoint still reads as broken.

With the rate no longer waiting on prose, the reasoning budget becomes a
sample size instead of room for an answer: 512, long enough that
per-request overhead does not colour the measurement and about 100 seconds
at 5 tok/s. The single retry for an undeclared thinking model stays, now
buying a fairer sample as much as a chance at an answer, and never trades
a measurement it already has for a failure.

`run_preflight` was already reading the rate rather than `probe["ok"]`;
that is now stated where it happens, because it is load-bearing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(frontend): say when a rate was measured without a finished answer (#507 ) d7a7e14422

On a self-hosted reasoning model this is the normal successful probe: a
real generation rate from a request that never got as far as a sentence.
Unexplained, it reads as a minute of waiting followed by half a result, so
the panel now says the number came from the tokens the model did produce
and that this is the same decoding a real summarisation pays for.

Only shown when there is a rate. A probe that measured nothing keeps the
"probe failed" figure and gets no reassurance.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs: the preflight measures throughput, not whether the model finished (#507 )

CI / Backend lint (ruff) (pull_request) Successful in 1m14s

Details

CI / Bot/backend version sync (pull_request) Successful in 24s

Details

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 1m56s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 2m28s

Details

CI / Bot tests and audit (pull_request) Successful in 2m53s

Details

CI / Docker image build (pull_request) Successful in 4m11s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 11m16s

Details

CI / Synthetic session harness (no GPU, no LLM) (pull_request) Successful in 14m55s

Details

1bc98d9d32

The reference table quoted the 2048-token reasoning budget and implied the
rate depended on getting an answer. Both changed: 512, and a local
reasoning model that spends the whole budget thinking is still measured.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed

2026-09-05 16:49:12 +00:00

claude-bot merged commit b38ab27de7 into main

2026-09-05 17:05:11 +00:00

claude-bot deleted branch fix/507-rate-from-exhausted-budget

2026-09-05 17:05:12 +00:00

claude-bot referenced this pull request from a commit

2026-09-05 17:05:12 +00:00

Merge PR #512: fix(backend): rate the LLM on what it decoded, not on whether it finished

claude-bot referenced this pull request

2026-09-05 17:06:02 +00:00

chore(release): v4.2.2 #513