Spike: does the LLM endpoint honour constrained decoding (json_schema / grammar)? #281
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Follow-up to #279. That issue shipped a tolerant parser — a safety net. This is the spike for the actual cure.
The problem it addresses
generate_structured_textsendsresponse_format: {"type":"json_object"}(webapp/backend/app/services/llm_service.py:174-183), and the endpoint accepts it without enforcing it. Measured against the live llama.cpp router (build b9029 fronting Qwen3.5-9B-UD-Q8_K_XL,--ctx-size 131072 --parallel 2): 6 of 9 prod-shaped calls returnedfinish_reason=stopwith structurally invalid JSON on a ~33k-token prompt. Compact single-line output correlated perfectly with malformation; pretty-printed output always parsed.v3.11.1 recovers from that and logs it. It does not stop it happening.
Questions to answer
response_format: {"type":"json_schema", …}get honoured by this router build? If yes, the whole failure class disappears for the llama.cpp path.grammarparameter work wherejson_schemadoesn't?Measure against real telemetry, not the 9-sample estimate
v3.11.1 added WARNING lines that make this countable for the first time:
Both log response size only, never content. Pull a real rate from the worker logs before and after any change rather than re-running a hand-rolled sample.
Portability constraint
generate_structured_textdispatches to four providers — Anthropic, OpenAI, Ollama, llama.cpp (llm_service.py:82-113). Agrammarparameter is llama.cpp-specific andjson_schemasupport varies. Whatever lands must degrade cleanly on providers that ignore it, which means the tolerant parser from #279 stays regardless — this would demote it from primary defence to safety net, not replace it.Do not regress
Keep
chat_template_kwargs: {"enable_thinking": False}underjson_mode(llm_service.py:183). Removing it whilemax_tokens=2048is set yieldsfinish_reason=lengthwith empty content — the entire budget goes to reasoning tokens. This was measured, and the existing comment documents it.Definition of done
A short written finding: which of
json_schema/grammarthe endpoint actually honours, whether the router is implicated, and a recommendation. Code only if the answer is favourable.Labels: backend
Picking this up. Early findings against the live router (
10.3.0.28:8090,qwen3.5,system_fingerprint b9029-2bacb1eb7— the same build the issue measured).Q1 — is
json_schemahonoured? Yes, and it is genuinely constraining.A schema whose field names no model would invent unprompted:
json_objectjson_schema['headcount_parity', 'nested', 'zzq_designation']That rules out "the model happened to guess the right shape" — the output tracked the schema, including the enum and the nested object. Worth noting the
json_objectcontrol produced malformed JSON on a tiny prompt, so the failure is not purely a large-context effect.Q2 — does an explicit
grammarparameter work? Yes. A hand-written GBNF root rule was obeyed exactly, emitting compact output matching the grammar. So both mechanisms are available;json_schemais the portable one and needs no GBNF authoring.Q3 — router or model? Moot, and that is the good outcome. The router is llama-swap fronting
llama-serveron127.0.0.1:40675(--ctx-size 131072 --parallel 2,Qwen3.5-9B-UD-Q8_K_XL). Since constrained decoding is honoured through the router, there is no router bug to chase — the direct-to-llama-server comparison was only needed ifjson_schemahad failed.Q4 — prompt-size sensitivity: sweep in progress (5k / 15k / 33k tokens, n=6 per cell,
json_objectvsjson_schema, production-shaped lore-proposal schema). Will post the table.On measuring from real telemetry: not currently possible. The prod worker restarted at 23:18 UTC and no lore work has run since, so
structurally invalidandnot parseableare both 0 — the log window has no qualifying calls rather than a clean bill of health. The numbers below are controlled probes, not observed production rate. Re-grepping after a few real sessions is still the right way to confirm any fix.Findings
All probes against
10.3.0.28:8090,qwen3.5,b9029-2bacb1eb7, production-shaped lore-proposal schema,max_tokens=2048unless stated.Q1 / Q2 — both mechanisms are honoured
json_schemagenuinely constrains decoding, proven with a schema whose field names no model would invent (zzq_designation,headcount_parityenum, nestedk7): output conformed exactly, while thejson_objectcontrol on the same prompt returned invalid JSON. An explicit GBNFgrammarwas also obeyed exactly.json_schemais the portable choice and needs no GBNF authoring.Q3 — the router is not implicated
llama-swap fronting
llama-serveron127.0.0.1:40675. Constrained decoding works through the router, so the direct-to-llama-server comparison is unnecessary — it was only needed hadjson_schemafailed.Q4 — prompt size: no reproducible effect
Valid JSON of the expected shape, raw
json.loads, n=6 per cell:json_objectjson_schemaTotals:
json_object24/26,json_schema26/26.The headline caveat: I could not reproduce the 6/9 rate
This issue records 6 of 9 prod-shaped calls returning
finish_reason=stopwith invalid JSON. I measured 2 of 26 (~8%) in the same class — an order of magnitude lower — and adding transcript entropy made it better, not worse (8/8 clean).Both failures I did see were the trivial variant:
{"proposals":[]}plus a stray closing brace, whichrepair_json_objectrecovers cleanly. So on my data the endpoint's misbehaviour is real but mild.I can't explain the gap from here. Candidates: real prompts carry approved-lore context I didn't synthesise; the original sample may have hit a different server state; or n=9 on a ~10% base rate is simply noisy. Treat the 2/3 figure as unconfirmed rather than as a measured production rate — which makes re-grepping the worker logs after a few real sessions the necessary next step, not an optional one.
What I did find:
max_tokenstruncation, andjson_schemadoes not fix itMy completions ran 300–1,200 tokens against a 2,048 cap — never near it. Forcing the cap to 256 on the same prompt:
json_objectfinish_reason=length→ invalid JSONjson_schemafinish_reason=length→ invalid JSONConstrained decoding guarantees the output follows the grammar; it cannot stop generation being cut off mid-structure. Both modes fail identically here.
Worse, this failure is currently silent.
repair_json_object"recovers" each truncated response into 2–3 proposals — dropping everything past the cut — and_structured_llamacppnever inspectsfinish_reasonwhen content is non-empty (llm_service.py:196-212; it only logs it on the empty-content path). A caller receiving 3 proposals cannot tell that from 3-of-12. This is precisely the semantic-incompleteness hazardrepair_json_object's own docstring warns about, reached through a path nothing currently detects.Recommendation
json_schema— proven, free, and it eliminates a class the tolerant parser cannot: wrong shape. Thejson_objectcontrol inventedtavern_name/patron_countfor a schema asking forname/patrons; a schema also enforces the 13-valueentry_typeenum and required fields. Add an optionaljson_schema: dict | Nonetogenerate_structured_textand let each adapter use it if it can — llama.cpp and OpenAI viaresponse_format, Ollama viaformat: <schema>, Anthropic falling back to today's{prefill. Degrades cleanly, per the portability constraint.finish_reason=length— likely the higher-value fix, and independent of everything above. Silently returning a repaired partial set is worse than raising. This wants its own issue.chat_template_kwargs.enable_thinking=Falsewas kept in every probe; no regression observed.Spike result: the endpoint honours constrained decoding — but not through the shape we were sending
Run against the live llama.cpp endpoint (
10.3.0.28:8090,qwen3.5) from the prod worker container. Read-only.The #339 claim was false as shipped. It said that on llama.cpp the schema becomes a decoding grammar so a small local model cannot emit malformed JSON. In production, every
json_schema=call was doing nothing at all.The probe
One adversarial prompt, fighting the schema on three axes at once — asks for prose instead of JSON, asks for a value outside an
enum, and asks for an extra field againstadditionalProperties: false. Only a real grammar can hold all three. Plus an impossible schema (dangling$ref), which distinguishes honoured from silently ignored: a server that compiles the schema must error on one it cannot compile.{"mood": "angry"}— enum honoured, no extra field{"mood": "angry"}{"type":"json_object"}Why
/propson this endpoint reports{"role":"router","max_instances":4,...}. It is not a vanillallama-server— it is a router in front of llama.cpp that implements the OpenAIresponse_format.json_schema.schemacontract, not llama.cpp's native flat one.llama.cpp's README documents the flat form, and #339 took it on that authority. That is right for a vanilla
llama-serverand wrong for what a self-hoster is often actually running. Since we cannot ask which is in front, both spellings are now sent — each server reads the key it knows and ignores the other. Both were verified to enforce, and both to 400 on an impossible schema.The uncomfortable part
Plain
{"type": "json_object"}is not enforced on this endpoint either. It returned prose for the same prompt. The tolerant repair parser (repair_json_object) has been carrying structured output on this deployment the whole time, which is exactly why nobody noticed — the failure was absorbed one layer down.That reframes the v4.0.0 premise a little. "The accuracy floor must not depend on model size" was resting partly on a guarantee that was not actually in force. With the corrected shape it now is, and the adversarial probe is the evidence.
And the process failure
The unit test pinning this shape passed the entire time. It asserted the flat form because that is what the code did, and both came from the same README paragraph. A unit test on a wire format can only ever confirm the belief that produced it.
The replacement names the endpoint the shape was verified against instead of citing documentation, and pins that the two spellings cannot drift apart. The general lesson, worth carrying into #349's eval harness: a contract with an external service needs at least one probe against the real thing, or the test suite is measuring our own assumptions back to us.
Not covered
format: <schema>— unverified, no endpoint to hand. Same class of risk; worth a probe before relying on it.llama-server— not reachable here, so whether it accepts the flat form was not confirmed directly. Sending both makes that moot.Fixed in
3b25451.Verified against the definition of done before closing.
This was a spike, and its deliverable was a finding: both
json_schemaandgrammarare honoured by the live llama.cpp router; the router was not implicated; no reproducible prompt-size effect; andmax_tokenstruncation defeats both modes identically (spun out separately).The finding was acted on — the code landed under #339 — and then the spike caught its own follow-up regression: the shape #339 shipped, taken from llama.cpp's README, was silently ignored by the production router, which implements the OpenAI contract. Verified adversarially with an impossible-schema probe (200 vs 400) and fixed in
3b25451.Current state sends both spellings simultaneously (
llm_service.py:418-447), pinned bytest_llamacpp_sends_the_schema_under_both_spellingsandtest_the_two_llamacpp_spellings_carry_the_same_schema(tests/test_llm_transport.py:232-284). Real callers wired atbeat_service.py:455andaudio_service.py:2567.Closing. Part of a full acceptance-criteria pass across the v4.0.0 milestone.