rbrooks/Quest-Board

Watch

Fork 0

[Backend] Replace the Anthropic assistant prefill with structured outputs #339

New issue

Closed

opened 2026-08-25 20:38:57 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-08-25 20:38:57 +00:00

Contributor

Severity: MEDIUM (blocker for hosted). Found in the August 2026 session lifecycle review (#319).

The defect

_structured_anthropic steers JSON output with an assistant-message prefill of "{" (llm_service.py:280-296). Assistant prefill was removed on Claude 4.6+ and the entire 5-family and now returns a 400.

The pinned default (claude-haiku-4-5-20251001) still accepts it, so this is latent — but any operator who configures a current model breaks every structured tool in the product at once, with an error that will read as a Quest Board bug. For a hosted tier that intends to use a modern Claude model, this is a hard blocker.

Proposed fix

Replace the prefill with the supported mechanism: output_config: {format: {...}} with a JSON schema on messages.create(). Note this is the current parameter — the older output_format top-level parameter is deprecated.

While in here, confirm the rest of the Anthropic path against current API behaviour rather than recalled behaviour, since several shapes changed across 2025-26.

Acceptance criteria

The "{" prefill is removed
Structured output uses output_config.format with a schema
A current-generation Claude model is exercised in tests or a documented manual check
The model id is configurable and documented, with a note that dated snapshots and current ids behave differently here

**Severity: MEDIUM (blocker for hosted).** Found in the August 2026 session lifecycle review (#319). ## The defect `_structured_anthropic` steers JSON output with an assistant-message prefill of `"{"` (`llm_service.py:280-296`). **Assistant prefill was removed on Claude 4.6+ and the entire 5-family and now returns a 400.** The pinned default (`claude-haiku-4-5-20251001`) still accepts it, so this is latent — but any operator who configures a current model breaks **every structured tool in the product** at once, with an error that will read as a Quest Board bug. For a hosted tier that intends to use a modern Claude model, this is a hard blocker. ## Proposed fix Replace the prefill with the supported mechanism: `output_config: {format: {...}}` with a JSON schema on `messages.create()`. Note this is the current parameter — the older `output_format` top-level parameter is deprecated. While in here, confirm the rest of the Anthropic path against current API behaviour rather than recalled behaviour, since several shapes changed across 2025-26. ## Acceptance criteria - [ ] The `"{"` prefill is removed - [ ] Structured output uses `output_config.format` with a schema - [ ] A current-generation Claude model is exercised in tests or a documented manual check - [ ] The model id is configurable and documented, with a note that dated snapshots and current ids behave differently here

claude-bot added this to the v4.0.0 - Session Pipeline Accuracy milestone

2026-08-25 20:38:57 +00:00

claude-bot added the

bug

backend

labels

2026-08-25 20:38:57 +00:00

rbrooks referenced this issue from a commit

2026-08-26 02:15:55 +00:00

fix(backend): replace the Anthropic prefill with real structured outputs (#339)

rbrooks referenced this issue from a commit

2026-08-26 14:52:43 +00:00

fix(backend): the llama.cpp json_schema shape #339 shipped did nothing (#281)

claude-bot referenced this issue

2026-08-26 14:53:01 +00:00

Spike: does the LLM endpoint honour constrained decoding (json_schema / grammar)? #281

rbrooks referenced this issue from a commit

2026-08-26 18:51:18 +00:00

fix(backend): stop sampling at random during structured extraction (#423)

claude-bot referenced this issue

2026-08-26 19:11:05 +00:00

[Backend] Summary quality varies 8x run-to-run on identical input #423

claude-bot referenced this issue

2026-08-27 23:26:57 +00:00

Spike: does the LLM endpoint honour constrained decoding (json_schema / grammar)? #281

claude-bot commented

2026-08-27 23:28:27 +00:00

Author

Contributor

Verified before closing. I went in expecting to find the prefill still lurking, given it was silently broken once. It is genuinely gone.

Prefill removed — no "{" literal and no role: "assistant" message anywhere in llm_service.py or audio_service.py. Only explanatory comments reference the old behaviour. Pinned by test_anthropic_no_longer_prefills_the_assistant_turn, which asserts roles == ["user"].
Structured output sent — body["output_config"] = {"format": {"type": "json_schema", "schema": schema}} (llm_service.py:405-408, sent from :672). Wire shape pinned by test_anthropic_sends_the_schema_as_output_config_format.
Model id configurable and documented — docs/OPERATIONS.md:96-121, including the dated-snapshot-versus-current-id note.

The note, and it matters given this issue's own history. The criterion asking that a current-generation model be exercised is met only by mocked tests. They name claude-opus-5, but the HTTP layer is stubbed — so what is proven is our request shape, not that Anthropic's endpoint accepts it. There is no live-API test infrastructure in the repo and no key available to check independently.

That is precisely the gap that made the original prefill bug latent: a unit test agreeing with the same belief that produced the code. It is also exactly what #281 later caught on the llama.cpp side — a documented request shape, shipped, silently ignored in production, with a green test the whole time.

So: closing, because the work asked for is done and the mocked tests are correct about what they cover. But the Anthropic path has never been confirmed against a real endpoint, and until someone runs one call with a real key, "verified" here means verified-by-construction. Worth doing before any hosted launch that offers Anthropic as a provider.

Part of a full acceptance-criteria pass across the v4.0.0 milestone.

Verified before closing. I went in expecting to find the prefill still lurking, given it was silently broken once. **It is genuinely gone.** - **Prefill removed** — no `"{"` literal and no `role: "assistant"` message anywhere in `llm_service.py` or `audio_service.py`. Only explanatory comments reference the old behaviour. Pinned by `test_anthropic_no_longer_prefills_the_assistant_turn`, which asserts `roles == ["user"]`. - **Structured output sent** — `body["output_config"] = {"format": {"type": "json_schema", "schema": schema}}` (`llm_service.py:405-408`, sent from `:672`). Wire shape pinned by `test_anthropic_sends_the_schema_as_output_config_format`. - **Model id configurable and documented** — `docs/OPERATIONS.md:96-121`, including the dated-snapshot-versus-current-id note. **The note, and it matters given this issue's own history.** The criterion asking that a current-generation model be exercised is met only by **mocked** tests. They name `claude-opus-5`, but the HTTP layer is stubbed — so what is proven is *our request shape*, not that Anthropic's endpoint accepts it. There is no live-API test infrastructure in the repo and no key available to check independently. That is precisely the gap that made the original prefill bug latent: a unit test agreeing with the same belief that produced the code. It is also exactly what #281 later caught on the llama.cpp side — a documented request shape, shipped, silently ignored in production, with a green test the whole time. So: closing, because the work asked for is done and the mocked tests are correct about what they cover. But **the Anthropic path has never been confirmed against a real endpoint**, and until someone runs one call with a real key, "verified" here means verified-by-construction. Worth doing before any hosted launch that offers Anthropic as a provider. Part of a full acceptance-criteria pass across the v4.0.0 milestone.

claude-bot closed this issue

2026-08-27 23:28:31 +00:00