LLM transport: graceful handling of reasoning models (thinking mode) #232

New issue

Closed

opened 2026-07-21 18:32:06 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-21 18:32:06 +00:00

Collaborator

Context

Self-hosters increasingly point Quest Board at reasoning models (Qwen3, DeepSeek-R1, gpt-oss, …) that emit hidden reasoning_content before any visible output. v3.10 added chat_template_kwargs={"enable_thinking": false} on json-mode llama.cpp calls (llm_service._structured_llamacpp, commit 07552a3), which fixes structured generation — every GM Workbench tool now works against a live Qwen3 server. This issue tracks the remaining rough edges so the experience degrades gracefully instead of opaquely.

Discovered while dev-testing v3.10 against a local qwen3.5 llama.cpp server (see PR #225 thread).

Rough edges to address

Admin → Bot Settings "Test LLM" is misleading for reasoning models.
audio_service._test_llm_llamacpp (and _test_llm_anthropic) cap the probe at max_tokens=16. A reasoning model spends all 16 on reasoning_content and returns empty visible content, so the Test button reports empty/blank even though the endpoint is perfectly healthy. Fix: raise the probe's max_tokens, and/or disable thinking in the test call, and/or detect reasoning_content and report success. Apply consistently across the ollama/openai/anthropic/llama.cpp test variants.
Opaque "empty response" on finish_reason='length'.
When a model exhausts its output budget on reasoning, _structured_llamacpp raises the generic "LLM returned an empty response … the model may not support json_object mode." For a reasoning model the real cause is token-budget exhaustion. Detect finish_reason == 'length' and surface an actionable message (e.g. "the model hit its output-token limit — if it is a reasoning model, disable thinking or raise the output limit").
Make thinking-disable configurable / broaden coverage.
Today enable_thinking=false is hardcoded to the json-mode llama.cpp path. Consider:
- an Admin toggle ("model uses reasoning — disable thinking") persisted in llm_config;
- the Ollama path (its /api/generate accepts a think boolean);
- prose flows (json_mode=false, e.g. campaign storyline) where a reasoning model can still overrun.

Scope & priority

Backend LLM transport + Admin Bot Settings UI. Backend-only, no BOT_CONTRACT_VERSION bump. P4 / backlog quality-of-life — the core v3.10 path already works without it. Non-blocking for the v3.10 release.

Ref: PR #225; commit 07552a3; llm_service.py, audio_service._test_llm_*.

🤖 Generated with Claude Code

## Context Self-hosters increasingly point Quest Board at **reasoning models** (Qwen3, DeepSeek-R1, gpt-oss, …) that emit hidden `reasoning_content` before any visible output. v3.10 added `chat_template_kwargs={"enable_thinking": false}` on **json-mode llama.cpp** calls (`llm_service._structured_llamacpp`, commit `07552a3`), which fixes structured generation — every GM Workbench tool now works against a live Qwen3 server. This issue tracks the remaining rough edges so the experience degrades gracefully instead of opaquely. Discovered while dev-testing v3.10 against a local `qwen3.5` llama.cpp server (see PR #225 thread). ## Rough edges to address 1. **Admin → Bot Settings "Test LLM" is misleading for reasoning models.** `audio_service._test_llm_llamacpp` (and `_test_llm_anthropic`) cap the probe at `max_tokens=16`. A reasoning model spends all 16 on `reasoning_content` and returns empty visible `content`, so the Test button reports empty/blank even though the endpoint is perfectly healthy. Fix: raise the probe's `max_tokens`, and/or disable thinking in the test call, and/or detect `reasoning_content` and report success. Apply consistently across the ollama/openai/anthropic/llama.cpp test variants. 2. **Opaque "empty response" on `finish_reason='length'`.** When a model exhausts its output budget on reasoning, `_structured_llamacpp` raises the generic *"LLM returned an empty response … the model may not support json_object mode."* For a reasoning model the real cause is token-budget exhaustion. Detect `finish_reason == 'length'` and surface an actionable message (e.g. *"the model hit its output-token limit — if it is a reasoning model, disable thinking or raise the output limit"*). 3. **Make thinking-disable configurable / broaden coverage.** Today `enable_thinking=false` is hardcoded to the json-mode llama.cpp path. Consider: - an Admin toggle ("model uses reasoning — disable thinking") persisted in `llm_config`; - the Ollama path (its `/api/generate` accepts a `think` boolean); - prose flows (`json_mode=false`, e.g. campaign storyline) where a reasoning model can still overrun. ## Scope & priority Backend LLM transport + Admin Bot Settings UI. Backend-only, no `BOT_CONTRACT_VERSION` bump. **P4 / backlog** quality-of-life — the core v3.10 path already works without it. Non-blocking for the v3.10 release. Ref: PR #225; commit `07552a3`; `llm_service.py`, `audio_service._test_llm_*`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added this to the Backlog - Speculative / Long-Term milestone

2026-07-21 18:33:17 +00:00

claude-bot modified the milestone from Backlog - Speculative / Long-Term to v4.0.0 - Session Pipeline Accuracy

2026-08-25 20:39:55 +00:00

claude-bot referenced this issue

2026-08-25 20:45:22 +00:00

[Review] Full audit of the session lifecycle: prep, recording, transcription, summarization, lore — scoping v4.0.0 #319

rbrooks referenced this issue from a commit

2026-08-26 18:51:18 +00:00

fix(backend): stop sampling at random during structured extraction (#423)

claude-bot referenced this issue

2026-08-26 19:11:05 +00:00

[Backend] Summary quality varies 8x run-to-run on identical input #423

claude-bot commented

2026-08-27 23:28:18 +00:00

Author

Collaborator

Verified before closing. Both reported bugs are fixed and tested; one "Consider:" item was not built, and this note is so that stays visible.

Fixed:

The test-LLM probe no longer reports success on a reasoning model — _TEST_LLM_MAX_TOKENS = 64 applied consistently, with per-provider hidden-reasoning detection: llama.cpp checks reasoning_content, Ollama checks thinking, OpenAI checks usage.completion_tokens_details.reasoning_tokens (audio_service.py:4050-4192). Anthropic is N/A on a bare call. Tested across all four (tests/test_llm_transport.py:492-588).
finish_reason='length' with empty content gives an actionable message — llm_service.py:605-619 says the budget likely went to hidden reasoning and names the remedy, instead of the generic empty-response error.
Ollama coverage — think: false sent whenever json_mode (llm_service.py:480-501).

Not built: the admin toggle ("this model uses reasoning") persisted in llm_config. Grepped settings_service.py and the whole frontend for reasoning/thinking — nothing. Thinking-disable is unconditional for json_mode rather than operator-controlled. The issue phrased this as "Consider:" rather than a requirement, and unconditional-for-structured-output is defensible, since structured extraction never benefits from reasoning tokens. But an operator with a model that behaves differently has no lever.

Deliberately not done, correctly: prose flows (json_mode=False) keep thinking on, with a code comment explaining that turning it off is a narrative-quality trade-off this module has no basis to make on the operator's behalf, and a test pinning it (test_llamacpp_prose_mode_omits_thinking_kwarg). That boundary later became the model for #423's sampling decision.

Closing. Part of a full acceptance-criteria pass across the v4.0.0 milestone.

Verified before closing. Both reported bugs are fixed and tested; one "Consider:" item was not built, and this note is so that stays visible. **Fixed:** - **The test-LLM probe no longer reports success on a reasoning model** — `_TEST_LLM_MAX_TOKENS = 64` applied consistently, with per-provider hidden-reasoning detection: llama.cpp checks `reasoning_content`, Ollama checks `thinking`, OpenAI checks `usage.completion_tokens_details.reasoning_tokens` (`audio_service.py:4050-4192`). Anthropic is N/A on a bare call. Tested across all four (`tests/test_llm_transport.py:492-588`). - **`finish_reason='length'` with empty content gives an actionable message** — `llm_service.py:605-619` says the budget likely went to hidden reasoning and names the remedy, instead of the generic empty-response error. - **Ollama coverage** — `think: false` sent whenever `json_mode` (`llm_service.py:480-501`). **Not built:** the admin toggle ("this model uses reasoning") persisted in `llm_config`. Grepped `settings_service.py` and the whole frontend for reasoning/thinking — nothing. Thinking-disable is unconditional for `json_mode` rather than operator-controlled. The issue phrased this as *"Consider:"* rather than a requirement, and unconditional-for-structured-output is defensible, since structured extraction never benefits from reasoning tokens. But an operator with a model that behaves differently has no lever. **Deliberately not done, correctly:** prose flows (`json_mode=False`) keep thinking on, with a code comment explaining that turning it off is a narrative-quality trade-off this module has no basis to make on the operator's behalf, and a test pinning it (`test_llamacpp_prose_mode_omits_thinking_kwarg`). That boundary later became the model for #423's sampling decision. Closing. Part of a full acceptance-criteria pass across the v4.0.0 milestone.

claude-bot closed this issue

2026-08-27 23:28:31 +00:00

rbrooks referenced this issue from a commit

2026-08-28 19:53:43 +00:00

chore(evals): record the v4.0.0 live run against the real session

claude-bot referenced this issue

2026-08-28 23:26:29 +00:00

v4.0.0 — deterministic attribution: the summary is composed from checked events, not asserted #434