rbrooks/Quest-Board

Watch

Fork 0

Spike: does the LLM endpoint honour constrained decoding (json_schema / grammar)? #281

New issue

Closed

opened 2026-08-05 19:46:56 +00:00 by claude-bot · 4 comments

claude-bot commented

2026-08-05 19:46:56 +00:00

Collaborator

Follow-up to #279. That issue shipped a tolerant parser — a safety net. This is the spike for the actual cure.

The problem it addresses

generate_structured_text sends response_format: {"type":"json_object"} (webapp/backend/app/services/llm_service.py:174-183), and the endpoint accepts it without enforcing it. Measured against the live llama.cpp router (build b9029 fronting Qwen3.5-9B-UD-Q8_K_XL, --ctx-size 131072 --parallel 2): 6 of 9 prod-shaped calls returned finish_reason=stop with structurally invalid JSON on a ~33k-token prompt. Compact single-line output correlated perfectly with malformation; pretty-printed output always parsed.

v3.11.1 recovers from that and logs it. It does not stop it happening.

Questions to answer

Does response_format: {"type":"json_schema", …} get honoured by this router build? If yes, the whole failure class disappears for the llama.cpp path.
Does an explicit llama.cpp grammar parameter work where json_schema doesn't?
Router or model? Hit the underlying llama.cpp server directly, bypassing the router, with an identical body. If the direct call enforces the grammar and the routed one doesn't, this is a router bug and the fix is infrastructure, not code.
Does prompt size matter independently? The corruption correlated with a large prompt. Worth testing the same schema at 5k / 15k / 33k tokens to see whether this is a context-length effect that would also be mitigated by chunking extraction (the lore pipeline already does multi-pass).

Measure against real telemetry, not the 9-sample estimate

v3.11.1 added WARNING lines that make this countable for the first time:

grep -c "structurally invalid" # recovered by repair
grep -c "not parseable"        # total failure

Both log response size only, never content. Pull a real rate from the worker logs before and after any change rather than re-running a hand-rolled sample.

Portability constraint

generate_structured_text dispatches to four providers — Anthropic, OpenAI, Ollama, llama.cpp (llm_service.py:82-113). A grammar parameter is llama.cpp-specific and json_schema support varies. Whatever lands must degrade cleanly on providers that ignore it, which means the tolerant parser from #279 stays regardless — this would demote it from primary defence to safety net, not replace it.

Do not regress

Keep chat_template_kwargs: {"enable_thinking": False} under json_mode (llm_service.py:183). Removing it while max_tokens=2048 is set yields finish_reason=length with empty content — the entire budget goes to reasoning tokens. This was measured, and the existing comment documents it.

Definition of done

A short written finding: which of json_schema / grammar the endpoint actually honours, whether the router is implicated, and a recommendation. Code only if the answer is favourable.

Labels: backend

Follow-up to #279. That issue shipped a **tolerant parser** — a safety net. This is the spike for the actual cure. ## The problem it addresses `generate_structured_text` sends `response_format: {"type":"json_object"}` (`webapp/backend/app/services/llm_service.py:174-183`), and the endpoint accepts it without enforcing it. Measured against the live llama.cpp router (build b9029 fronting Qwen3.5-9B-UD-Q8_K_XL, `--ctx-size 131072 --parallel 2`): **6 of 9 prod-shaped calls returned `finish_reason=stop` with structurally invalid JSON** on a ~33k-token prompt. Compact single-line output correlated perfectly with malformation; pretty-printed output always parsed. v3.11.1 recovers from that and logs it. It does not stop it happening. ## Questions to answer 1. **Does `response_format: {"type":"json_schema", …}` get honoured** by this router build? If yes, the whole failure class disappears for the llama.cpp path. 2. **Does an explicit llama.cpp `grammar` parameter work** where `json_schema` doesn't? 3. **Router or model?** Hit the underlying llama.cpp server directly, bypassing the router, with an identical body. If the direct call enforces the grammar and the routed one doesn't, this is a router bug and the fix is infrastructure, not code. 4. **Does prompt size matter independently?** The corruption correlated with a large prompt. Worth testing the same schema at 5k / 15k / 33k tokens to see whether this is a context-length effect that would also be mitigated by chunking extraction (the lore pipeline already does multi-pass). ## Measure against real telemetry, not the 9-sample estimate v3.11.1 added WARNING lines that make this countable for the first time: ``` grep -c "structurally invalid" # recovered by repair grep -c "not parseable" # total failure ``` Both log response **size only**, never content. Pull a real rate from the worker logs before and after any change rather than re-running a hand-rolled sample. ## Portability constraint `generate_structured_text` dispatches to **four** providers — Anthropic, OpenAI, Ollama, llama.cpp (`llm_service.py:82-113`). A `grammar` parameter is llama.cpp-specific and `json_schema` support varies. Whatever lands must degrade cleanly on providers that ignore it, which means the tolerant parser from #279 stays regardless — this would demote it from primary defence to safety net, not replace it. ## Do not regress Keep `chat_template_kwargs: {"enable_thinking": False}` under `json_mode` (`llm_service.py:183`). Removing it while `max_tokens=2048` is set yields `finish_reason=length` with empty content — the entire budget goes to reasoning tokens. This was measured, and the existing comment documents it. ## Definition of done A short written finding: which of `json_schema` / `grammar` the endpoint actually honours, whether the router is implicated, and a recommendation. Code only if the answer is favourable. Labels: backend

claude-bot added the

backend

label

2026-08-05 19:47:00 +00:00

claude-bot commented

2026-08-07 00:05:58 +00:00

Author

Collaborator

Picking this up. Early findings against the live router (10.3.0.28:8090, qwen3.5, system_fingerprint b9029-2bacb1eb7 — the same build the issue measured).

Q1 — is json_schema honoured? Yes, and it is genuinely constraining.

A schema whose field names no model would invent unprompted:

{"zzq_designation": "...", "headcount_parity": "odd|even", "nested": {"k7": 0}}

mode	result on the same 256-token prompt
`json_object`	invalid JSON
`json_schema`	exact conformance — `['headcount_parity', 'nested', 'zzq_designation']`

That rules out "the model happened to guess the right shape" — the output tracked the schema, including the enum and the nested object. Worth noting the json_object control produced malformed JSON on a tiny prompt, so the failure is not purely a large-context effect.

Q2 — does an explicit grammar parameter work? Yes. A hand-written GBNF root rule was obeyed exactly, emitting compact output matching the grammar. So both mechanisms are available; json_schema is the portable one and needs no GBNF authoring.

Q3 — router or model? Moot, and that is the good outcome. The router is llama-swap fronting llama-server on 127.0.0.1:40675 (--ctx-size 131072 --parallel 2, Qwen3.5-9B-UD-Q8_K_XL). Since constrained decoding is honoured through the router, there is no router bug to chase — the direct-to-llama-server comparison was only needed if json_schema had failed.

Q4 — prompt-size sensitivity: sweep in progress (5k / 15k / 33k tokens, n=6 per cell, json_object vs json_schema, production-shaped lore-proposal schema). Will post the table.

On measuring from real telemetry: not currently possible. The prod worker restarted at 23:18 UTC and no lore work has run since, so structurally invalid and not parseable are both 0 — the log window has no qualifying calls rather than a clean bill of health. The numbers below are controlled probes, not observed production rate. Re-grepping after a few real sessions is still the right way to confirm any fix.

Picking this up. Early findings against the live router (`10.3.0.28:8090`, `qwen3.5`, `system_fingerprint b9029-2bacb1eb7` — the same build the issue measured). **Q1 — is `json_schema` honoured? Yes, and it is genuinely constraining.** A schema whose field names no model would invent unprompted: ```json {"zzq_designation": "...", "headcount_parity": "odd|even", "nested": {"k7": 0}} ``` | mode | result on the same 256-token prompt | |---|---| | `json_object` | **invalid JSON** | | `json_schema` | exact conformance — `['headcount_parity', 'nested', 'zzq_designation']` | That rules out "the model happened to guess the right shape" — the output tracked the schema, including the enum and the nested object. Worth noting the `json_object` control produced malformed JSON on a *tiny* prompt, so the failure is not purely a large-context effect. **Q2 — does an explicit `grammar` parameter work? Yes.** A hand-written GBNF root rule was obeyed exactly, emitting compact output matching the grammar. So both mechanisms are available; `json_schema` is the portable one and needs no GBNF authoring. **Q3 — router or model? Moot, and that is the good outcome.** The router is llama-swap fronting `llama-server` on `127.0.0.1:40675` (`--ctx-size 131072 --parallel 2`, `Qwen3.5-9B-UD-Q8_K_XL`). Since constrained decoding *is* honoured **through** the router, there is no router bug to chase — the direct-to-llama-server comparison was only needed if `json_schema` had failed. **Q4 — prompt-size sensitivity:** sweep in progress (5k / 15k / 33k tokens, n=6 per cell, `json_object` vs `json_schema`, production-shaped lore-proposal schema). Will post the table. **On measuring from real telemetry:** not currently possible. The prod worker restarted at 23:18 UTC and no lore work has run since, so `structurally invalid` and `not parseable` are both **0** — the log window has no qualifying calls rather than a clean bill of health. The numbers below are controlled probes, not observed production rate. Re-grepping after a few real sessions is still the right way to confirm any fix.

claude-bot commented

2026-08-07 00:26:03 +00:00

Author

Collaborator

Findings

All probes against 10.3.0.28:8090, qwen3.5, b9029-2bacb1eb7, production-shaped lore-proposal schema, max_tokens=2048 unless stated.

Q1 / Q2 — both mechanisms are honoured

json_schema genuinely constrains decoding, proven with a schema whose field names no model would invent (zzq_designation, headcount_parity enum, nested k7): output conformed exactly, while the json_object control on the same prompt returned invalid JSON. An explicit GBNF grammar was also obeyed exactly. json_schema is the portable choice and needs no GBNF authoring.

Q3 — the router is not implicated

llama-swap fronting llama-server on 127.0.0.1:40675. Constrained decoding works through the router, so the direct-to-llama-server comparison is unnecessary — it was only needed had json_schema failed.

Q4 — prompt size: no reproducible effect

Valid JSON of the expected shape, raw json.loads, n=6 per cell:

prompt tokens	`json_object`	`json_schema`
5,000	6/6	6/6
15,000	5/6	6/6
33,000	5/6	6/6
33,000 (high-entropy)	8/8	8/8

Totals: json_object 24/26, json_schema 26/26.

The headline caveat: I could not reproduce the 6/9 rate

This issue records 6 of 9 prod-shaped calls returning finish_reason=stop with invalid JSON. I measured 2 of 26 (~8%) in the same class — an order of magnitude lower — and adding transcript entropy made it better, not worse (8/8 clean).

Both failures I did see were the trivial variant: {"proposals":[]} plus a stray closing brace, which repair_json_object recovers cleanly. So on my data the endpoint's misbehaviour is real but mild.

I can't explain the gap from here. Candidates: real prompts carry approved-lore context I didn't synthesise; the original sample may have hit a different server state; or n=9 on a ~10% base rate is simply noisy. Treat the 2/3 figure as unconfirmed rather than as a measured production rate — which makes re-grepping the worker logs after a few real sessions the necessary next step, not an optional one.

What I did find: `max_tokens` truncation, and `json_schema` does not fix it

My completions ran 300–1,200 tokens against a 2,048 cap — never near it. Forcing the cap to 256 on the same prompt:

mode	result (n=4)
`json_object`	3/4 `finish_reason=length` → invalid JSON
`json_schema`	2/4 `finish_reason=length` → invalid JSON

Constrained decoding guarantees the output follows the grammar; it cannot stop generation being cut off mid-structure. Both modes fail identically here.

Worse, this failure is currently silent. repair_json_object "recovers" each truncated response into 2–3 proposals — dropping everything past the cut — and _structured_llamacpp never inspects finish_reason when content is non-empty (llm_service.py:196-212; it only logs it on the empty-content path). A caller receiving 3 proposals cannot tell that from 3-of-12. This is precisely the semantic-incompleteness hazard repair_json_object's own docstring warns about, reached through a path nothing currently detects.

Recommendation

Adopt json_schema — proven, free, and it eliminates a class the tolerant parser cannot: wrong shape. The json_object control invented tavern_name/patron_count for a schema asking for name/patrons; a schema also enforces the 13-value entry_type enum and required fields. Add an optional json_schema: dict | None to generate_structured_text and let each adapter use it if it can — llama.cpp and OpenAI via response_format, Ollama via format: <schema>, Anthropic falling back to today's { prefill. Degrades cleanly, per the portability constraint.
But do not expect it to fix the reported symptom. It removes ~8% malformation on my data, not 67%, and gives zero protection against the truncation mode I could reproduce.
Detect finish_reason=length — likely the higher-value fix, and independent of everything above. Silently returning a repaired partial set is worse than raising. This wants its own issue.
Keep the tolerant parser regardless, exactly as this issue anticipated — demoted to safety net.

chat_template_kwargs.enable_thinking=False was kept in every probe; no regression observed.

## Findings All probes against `10.3.0.28:8090`, `qwen3.5`, `b9029-2bacb1eb7`, production-shaped lore-proposal schema, `max_tokens=2048` unless stated. ### Q1 / Q2 — both mechanisms are honoured `json_schema` genuinely constrains decoding, proven with a schema whose field names no model would invent (`zzq_designation`, `headcount_parity` enum, nested `k7`): output conformed exactly, while the `json_object` control on the same prompt returned **invalid JSON**. An explicit GBNF `grammar` was also obeyed exactly. `json_schema` is the portable choice and needs no GBNF authoring. ### Q3 — the router is not implicated llama-swap fronting `llama-server` on `127.0.0.1:40675`. Constrained decoding works *through* the router, so the direct-to-llama-server comparison is unnecessary — it was only needed had `json_schema` failed. ### Q4 — prompt size: no reproducible effect Valid JSON of the expected shape, raw `json.loads`, n=6 per cell: | prompt tokens | `json_object` | `json_schema` | |---|---|---| | 5,000 | 6/6 | 6/6 | | 15,000 | 5/6 | 6/6 | | 33,000 | 5/6 | 6/6 | | 33,000 (high-entropy) | 8/8 | 8/8 | **Totals: `json_object` 24/26, `json_schema` 26/26.** ### The headline caveat: I could not reproduce the 6/9 rate This issue records **6 of 9** prod-shaped calls returning `finish_reason=stop` with invalid JSON. I measured **2 of 26** (~8%) in the same class — an order of magnitude lower — and adding transcript entropy made it *better*, not worse (8/8 clean). Both failures I did see were the trivial variant: `{"proposals":[]}` plus a stray closing brace, which `repair_json_object` recovers cleanly. So on my data the endpoint's misbehaviour is real but mild. I can't explain the gap from here. Candidates: real prompts carry approved-lore context I didn't synthesise; the original sample may have hit a different server state; or n=9 on a ~10% base rate is simply noisy. **Treat the 2/3 figure as unconfirmed** rather than as a measured production rate — which makes re-grepping the worker logs after a few real sessions the necessary next step, not an optional one. ### What I did find: `max_tokens` truncation, and `json_schema` does not fix it My completions ran 300–1,200 tokens against a 2,048 cap — never near it. Forcing the cap to 256 on the same prompt: | mode | result (n=4) | |---|---| | `json_object` | 3/4 `finish_reason=length` → **invalid JSON** | | `json_schema` | 2/4 `finish_reason=length` → **invalid JSON** | Constrained decoding guarantees the output *follows* the grammar; it cannot stop generation being cut off mid-structure. **Both modes fail identically here.** Worse, this failure is currently silent. `repair_json_object` "recovers" each truncated response into 2–3 proposals — dropping everything past the cut — and `_structured_llamacpp` never inspects `finish_reason` when content is non-empty (`llm_service.py:196-212`; it only logs it on the empty-content path). A caller receiving 3 proposals cannot tell that from 3-of-12. This is precisely the semantic-incompleteness hazard `repair_json_object`'s own docstring warns about, reached through a path nothing currently detects. ## Recommendation 1. **Adopt `json_schema`** — proven, free, and it eliminates a class the tolerant parser cannot: wrong *shape*. The `json_object` control invented `tavern_name`/`patron_count` for a schema asking for `name`/`patrons`; a schema also enforces the 13-value `entry_type` enum and required fields. Add an optional `json_schema: dict | None` to `generate_structured_text` and let each adapter use it if it can — llama.cpp and OpenAI via `response_format`, Ollama via `format: <schema>`, Anthropic falling back to today's `{` prefill. Degrades cleanly, per the portability constraint. 2. **But do not expect it to fix the reported symptom.** It removes ~8% malformation on my data, not 67%, and gives **zero** protection against the truncation mode I could reproduce. 3. **Detect `finish_reason=length`** — likely the higher-value fix, and independent of everything above. Silently returning a repaired partial set is worse than raising. This wants its own issue. 4. **Keep the tolerant parser** regardless, exactly as this issue anticipated — demoted to safety net. `chat_template_kwargs.enable_thinking=False` was kept in every probe; no regression observed.

claude-bot referenced this issue

2026-08-07 00:26:38 +00:00

LLM transport: a truncated response (finish_reason=length) is silently repaired into a partial result #293

claude-bot referenced this issue

2026-08-25 19:32:48 +00:00

[Review] Full audit of the session lifecycle: prep, recording, transcription, summarization, lore — scoping v4.0.0 #319

claude-bot referenced this issue

2026-08-25 20:38:51 +00:00

[Backend] Emit structured, timestamped beats with evidence references #332

claude-bot referenced this issue

2026-08-25 20:39:14 +00:00

[Backend] Define the LLM provider contract with declared context window and schema output #351

claude-bot added this to the v4.0.0 - Session Pipeline Accuracy milestone

2026-08-25 20:39:53 +00:00

claude-bot referenced this issue

2026-08-25 20:45:22 +00:00

[Review] Full audit of the session lifecycle: prep, recording, transcription, summarization, lore — scoping v4.0.0 #319

rbrooks referenced this issue from a commit

2026-08-26 14:52:43 +00:00

fix(backend): the llama.cpp json_schema shape #339 shipped did nothing (#281)

claude-bot commented

2026-08-26 14:53:01 +00:00

Author

Collaborator

Spike result: the endpoint honours constrained decoding — but not through the shape we were sending

Run against the live llama.cpp endpoint (10.3.0.28:8090, qwen3.5) from the prod worker container. Read-only.

The #339 claim was false as shipped. It said that on llama.cpp the schema becomes a decoding grammar so a small local model cannot emit malformed JSON. In production, every json_schema= call was doing nothing at all.

The probe

One adversarial prompt, fighting the schema on three axes at once — asks for prose instead of JSON, asks for a value outside an enum, and asks for an extra field against additionalProperties: false. Only a real grammar can hold all three. Plus an impossible schema (dangling $ref), which distinguishes honoured from silently ignored: a server that compiles the schema must error on one it cannot compile.

Wire shape	Adversarial prompt	Impossible schema
flat — what #339 shipped	English prose, no JSON	HTTP 200 — never parsed
OpenAI-nested	`{"mood": "angry"}` — enum honoured, no extra field	HTTP 400 — grammar compilation failed
both keys	`{"mood": "angry"}`	HTTP 400
plain `{"type":"json_object"}`	English prose, no JSON	—

Why

/props on this endpoint reports {"role":"router","max_instances":4,...}. It is not a vanilla llama-server — it is a router in front of llama.cpp that implements the OpenAI response_format.json_schema.schema contract, not llama.cpp's native flat one.

llama.cpp's README documents the flat form, and #339 took it on that authority. That is right for a vanilla llama-server and wrong for what a self-hoster is often actually running. Since we cannot ask which is in front, both spellings are now sent — each server reads the key it knows and ignores the other. Both were verified to enforce, and both to 400 on an impossible schema.

The uncomfortable part

Plain {"type": "json_object"} is not enforced on this endpoint either. It returned prose for the same prompt. The tolerant repair parser (repair_json_object) has been carrying structured output on this deployment the whole time, which is exactly why nobody noticed — the failure was absorbed one layer down.

That reframes the v4.0.0 premise a little. "The accuracy floor must not depend on model size" was resting partly on a guarantee that was not actually in force. With the corrected shape it now is, and the adversarial probe is the evidence.

And the process failure

The unit test pinning this shape passed the entire time. It asserted the flat form because that is what the code did, and both came from the same README paragraph. A unit test on a wire format can only ever confirm the belief that produced it.

The replacement names the endpoint the shape was verified against instead of citing documentation, and pins that the two spellings cannot drift apart. The general lesson, worth carrying into #349's eval harness: a contract with an external service needs at least one probe against the real thing, or the test suite is measuring our own assumptions back to us.

Not covered

Ollama's format: <schema> — unverified, no endpoint to hand. Same class of risk; worth a probe before relying on it.
Vanilla (non-routed) llama-server — not reachable here, so whether it accepts the flat form was not confirmed directly. Sending both makes that moot.
Hosted OpenAI/Anthropic shapes — not exercised.

Fixed in 3b25451.

## Spike result: the endpoint honours constrained decoding — but not through the shape we were sending Run against the live llama.cpp endpoint (`10.3.0.28:8090`, `qwen3.5`) from the prod worker container. Read-only. **The #339 claim was false as shipped.** It said that on llama.cpp the schema becomes a decoding grammar so a small local model *cannot* emit malformed JSON. In production, every `json_schema=` call was doing nothing at all. ### The probe One adversarial prompt, fighting the schema on three axes at once — asks for prose instead of JSON, asks for a value outside an `enum`, and asks for an extra field against `additionalProperties: false`. Only a real grammar can hold all three. Plus an impossible schema (dangling `$ref`), which distinguishes *honoured* from *silently ignored*: a server that compiles the schema must error on one it cannot compile. | Wire shape | Adversarial prompt | Impossible schema | |---|---|---| | **flat — what #339 shipped** | English prose, no JSON | **HTTP 200** — never parsed | | **OpenAI-nested** | `{"mood": "angry"}` — enum honoured, no extra field | **HTTP 400** — grammar compilation failed | | **both keys** | `{"mood": "angry"}` | **HTTP 400** | | plain `{"type":"json_object"}` | English prose, no JSON | — | ### Why `/props` on this endpoint reports `{"role":"router","max_instances":4,...}`. It is not a vanilla `llama-server` — it is a router in front of llama.cpp that implements the **OpenAI** `response_format.json_schema.schema` contract, not llama.cpp's native flat one. llama.cpp's README documents the flat form, and #339 took it on that authority. That is right for a vanilla `llama-server` and wrong for what a self-hoster is often actually running. Since we cannot ask which is in front, **both spellings are now sent** — each server reads the key it knows and ignores the other. Both were verified to enforce, and both to 400 on an impossible schema. ### The uncomfortable part **Plain `{"type": "json_object"}` is not enforced on this endpoint either.** It returned prose for the same prompt. The tolerant repair parser (`repair_json_object`) has been carrying structured output on this deployment the whole time, which is exactly why nobody noticed — the failure was absorbed one layer down. That reframes the v4.0.0 premise a little. "The accuracy floor must not depend on model size" was resting partly on a guarantee that was not actually in force. With the corrected shape it now is, and the adversarial probe is the evidence. ### And the process failure The unit test pinning this shape **passed the entire time**. It asserted the flat form because that is what the code did, and both came from the same README paragraph. A unit test on a wire format can only ever confirm the belief that produced it. The replacement names the endpoint the shape was verified against instead of citing documentation, and pins that the two spellings cannot drift apart. The general lesson, worth carrying into #349's eval harness: a contract with an external service needs at least one probe against the real thing, or the test suite is measuring our own assumptions back to us. ### Not covered - **Ollama's `format: <schema>`** — unverified, no endpoint to hand. Same class of risk; worth a probe before relying on it. - **Vanilla (non-routed) `llama-server`** — not reachable here, so whether it accepts the flat form was not confirmed directly. Sending both makes that moot. - Hosted OpenAI/Anthropic shapes — not exercised. Fixed in `3b25451`.

claude-bot commented

2026-08-27 23:26:57 +00:00

Author

Collaborator

Verified against the definition of done before closing.

This was a spike, and its deliverable was a finding: both json_schema and grammar are honoured by the live llama.cpp router; the router was not implicated; no reproducible prompt-size effect; and max_tokens truncation defeats both modes identically (spun out separately).

The finding was acted on — the code landed under #339 — and then the spike caught its own follow-up regression: the shape #339 shipped, taken from llama.cpp's README, was silently ignored by the production router, which implements the OpenAI contract. Verified adversarially with an impossible-schema probe (200 vs 400) and fixed in 3b25451.

Current state sends both spellings simultaneously (llm_service.py:418-447), pinned by test_llamacpp_sends_the_schema_under_both_spellings and test_the_two_llamacpp_spellings_carry_the_same_schema (tests/test_llm_transport.py:232-284). Real callers wired at beat_service.py:455 and audio_service.py:2567.

Closing. Part of a full acceptance-criteria pass across the v4.0.0 milestone.

Verified against the definition of done before closing. This was a spike, and its deliverable was a finding: both `json_schema` and `grammar` are honoured by the live llama.cpp router; the router was not implicated; no reproducible prompt-size effect; and `max_tokens` truncation defeats both modes identically (spun out separately). The finding was acted on — the code landed under #339 — and then the spike **caught its own follow-up regression**: the shape #339 shipped, taken from llama.cpp's README, was silently ignored by the production router, which implements the OpenAI contract. Verified adversarially with an impossible-schema probe (200 vs 400) and fixed in `3b25451`. Current state sends both spellings simultaneously (`llm_service.py:418-447`), pinned by `test_llamacpp_sends_the_schema_under_both_spellings` and `test_the_two_llamacpp_spellings_carry_the_same_schema` (`tests/test_llm_transport.py:232-284`). Real callers wired at `beat_service.py:455` and `audio_service.py:2567`. Closing. Part of a full acceptance-criteria pass across the v4.0.0 milestone.

claude-bot closed this issue

2026-08-27 23:27:18 +00:00

claude-bot referenced this issue

2026-08-27 23:28:27 +00:00

[Backend] Replace the Anthropic assistant prefill with structured outputs #339

rbrooks referenced this issue from a commit

2026-08-28 18:39:30 +00:00

fix(backend): make the beat validator actually check what it claims (#332, #333)

claude-bot referenced this issue

2026-08-28 19:14:22 +00:00

[Backend] Emit structured, timestamped beats with evidence references #332

claude-bot referenced this issue