Quote board silently empty: strict JSON parse rejects ~2/3 of highlight responses, failure is indistinguishable from "no quotes found" #279
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The player-facing quote board (#116) has produced zero rows on prod. Root cause is a strict JSON parse rejecting malformed model output, with a silent-failure path that made it look like the model simply found nothing memorable.
Distinct from #278 (which is the publication gate on
approve_audio). This is about rows never being written in the first place.Mechanism
llm_service.py:174-183sendsresponse_format: {"type":"json_object"}. The prod endpoint (llama.cpp router build b9029 fronting Qwen3.5-9B-UD-Q8_K_XL) does not enforce the JSON grammar — it returnsfinish_reason=stopwith structurally invalid JSON.}, missing]}, and an extra trailing}._parse_highlights_json(webapp/backend/app/services/audio_service.py:963-990) is strict.json.loadsfails, and the only salvage path is gated behindif not text.startswith("{")(:980) — so for a response that starts with{, the repair never runs at all. Verified by reading: a truncated-but-{-leading response goes straight tojson.loads→except→{}.audio_service.py:1041-1043: empty dict →return [].reminder_tasks.py:2133-2137logsINFO … extracted 0 highlights— identical to the legitimate "model found nothing" outcome. No warning, no error.Evidence
Reproduced against the live endpoint with the real 81,302-char transcript from session
d633ad3c: 9 prod-shaped calls, 6 returned structurally invalid JSON (~67% failure rate). Correlation was perfect — every compact-style response was malformed, every pretty-printed one parsed.Live worker log for that session:
HTTP 200, 14.5s, no exception logged. The block runs; the parse is what fails.
Forcing a parseable response showed 5/5 quotes passed the verbatim/hallucination substring check plus 3 moments — 8 rows would have been written. The filter is not the blocker.
Correcting an earlier assumption
The symptom was originally read as "zero highlights across 6 transcribed sessions." In fact only one session has ever attempted extraction: #116 landed in
86e6e66on 2026-07-17 and the prod worker was rebuilt 2026-07-27, so the 2026-07-29 session is the only one whoseprocess_audioran the extraction block. The other transcripts (2026-03-25 → 2026-07-15) predate the feature entirely. So this is 1 failure out of 1 attempt against a measured ~67% failure rate — not 6 for 6.Ruled out
highlights_in_discordonly governs Discord posting.max_quotesresolves correctly from the DB (both campaigns: 5).:2069-2071vs:2109-2111).:2138— that path logs a WARNING; zero such lines in the full 39,884-line worker log.max_tokens=2048truncation —finish_reason=stopon every malformed run, completion tokens 229-382.Why summarisation works and this doesn't
_summarise_llamacpp(audio_service.py:2098-2129) sends noresponse_format, nomax_tokens, and parses nothing — it returnscontent.strip()as prose. There is no JSON contract to violate.The shared helper has the same flaw
extract_json_object(webapp/backend/app/services/llm_service.py:283-303) also fails on a missing trailing}—rfind("}")can't recover a truncated tail. Everyjson_mode=Truecaller is exposed to the same endpoint behaviour: lore proposals,generation_service, stat-block generation. Those raise rather than silently returning[], which is why only highlights failed quietly — but they are equally fragile. Worth fixing once, centrally, rather than per-caller.Fix direction
if not text.startswith("{")gate so salvage always runs; usejson.JSONDecoder().raw_decode()(fixes the extra-trailing-}case outright); add bracket-balance repair (walk the string tracking depth outside string literals, truncate to the last complete element, append missing closers). All 6 captured malformed samples are recoverable this way.{}, and have the caller distinguish "parse failed" from "0 highlights". The indistinguishable INFO line is what hid this for a full release cycle.json_object. Constrained decoding viaresponse_format: {"type":"json_schema", …}or an explicitgrammarwould eliminate the class entirely. Worth testing against this router build.process_audio(reminder_tasks.py:2105), whose audio is already deleted. A GM-facing "regenerate highlights from transcript" action would let existing sessions be backfilled.webapp/backend/tests/test_highlights.py:100-104only feeds well-formed JSON or total garbage. Add fixtures for the three real corruption shapes.Do not drop
chat_template_kwargs: {"enable_thinking": False}in any fix — testing showed that removing it while keepingmax_tokens=2048yieldsfinish_reason=lengthwith empty content, as the whole budget goes to reasoning tokens.Not verified
Why the router build lets ungrammatical JSON through (the GPU host's llama.cpp logs weren't reachable). Retry-only is insufficient — at a 67% failure rate, retry-once still fails ~45% of the time, so the repair is the necessary part.
Labels: bug, backend
Fixed and deployed in v3.11.1 (PR #280, merged as
01c9f86).What shipped
repair_json_object()inllm_serviceis now the shared tolerant parser used byextract_json_objectand_parse_highlights_json. It cascades: clean parse →raw_decode(which alone fixes the extra-brace shape) → close the open bracket stack → truncate to the last parseable point and close. Every candidate is validated withjson.loadsbefore being returned, so it can only ever return real JSON.Two properties, both arrived at by getting them wrong first and being caught by tests:
extract_highlightsdoes, and there is a test proving it drops a repaired half-object.strict=Trueexists for callers that can't tolerate this.{out of a bare[...], silently dropping every later element and presenting the fragment as the whole response.test_workbench_rumor::test_parse_rumor_output_handles_malformed_replycaught it in the full-suite run. A valid non-object now reports failure so the caller's own shape-coercion fallback runs, as before.Both repair and total failure now log at WARNING with response size only — never content, which is table talk. That was the actual reason this survived a release:
INFO … extracted 0 highlightswas indistinguishable from "the model found nothing memorable".Also fixed the adjacent data-loss bug: the delete-then-reinsert cleared existing highlights before checking whether extraction returned anything, so every retry of a failing session destroyed a good set from an earlier successful run.
Tests
New
tests/test_json_repair.py(15 tests) plus 5 intest_highlights.py, built on the three corruption shapes captured verbatim from the live endpoint. Full backend suite: 836 passed (up from 815).Still open — worth a separate issue
This fixes the parsing, not the endpoint. The llama.cpp router still is not enforcing its JSON grammar and will keep returning malformed responses at roughly the measured ~2/3 rate on large prompts; the difference is that it now recovers and says so.
The follow-up worth doing is the constrained-decoding spike: whether
response_format: {"type":"json_schema", …}or an explicitgrammaris honoured by this build, and whether hitting the underlying llama.cpp server directly (bypassing the router) behaves differently — that would distinguish a router problem from a model problem. The new WARNING lines give a real failure rate to measure against, which the 9-sample estimate could not.Closing this one; the spike should be filed on its own.
extractingforever #287extractingforever #287restart: on-failurein docker-compose.yml —unless-stoppedbypasses docker-host's boot pacing #306