[Backend] Summary quality varies 8x run-to-run on identical input #423
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found by the #349 eval harness on its first real measurement, immediately after fixing the output cap in
3bf85b3.The measurement
Three consecutive runs. Same transcript (a real 77-minute session, 1,529 lines), same model (qwen3.5-9B via llama.cpp at 131k context), same prompt, same code. Recorded in
webapp/backend/evals/results/history.jsonl.Coverage — the fraction of eleven hand-verified events that survive into the summary — ranges from 1 of 11 to 8 of 11. Beat validation ranges from 24% to 82%.
Run 1 is not merely a worse summary. It covered so few events that chronology could not be scored at all: there were fewer than two events to put in an order.
Why this matters more than the average
The milestone's stated goal is that "the accuracy floor must not depend on model size — self-hosted small models are a first-class target". These numbers say the floor currently depends on luck, which is worse, because it is invisible. A GM whose session lands on a run-1 draw gets a summary missing ten of eleven events, with nothing anywhere reporting that anything went wrong — the pipeline completed successfully.
It also undermines the thing #349 was built for. "We changed the prompt and it got better" cannot be claimed from a single run when single runs vary this much. Every future comparison needs n runs and a spread, not a number.
What is not the cause
3bf85b3.json_modealready sendsenable_thinking: False.Worth investigating
Acceptance criteria
python -m evalsgrows a repeat-count option so variance is a first-class output, not something you discover by running it three times by handReproducing
Needs
QB_EVAL_ENDPOINT(base URL — the client appends/v1/chat/completions),QB_EVAL_MODEL,QB_EVAL_CONTEXT_TOKENS, andALLOW_PRIVATE_SERVICE_URLS=truefor a private endpoint. The private fixture is not in git; seewebapp/backend/evals/README.md.Dominant cause found and fixed in
0002fa4;--repeatadded in37f8c72. Leaving this open — two criteria are met, two are not.The cause: we were sampling at random
llm_servicesent no sampling parameters at all — notemperature,top_p,top_korseed— so every structured extraction ran at the server's default, which is tuned for creative writing. Extraction against a fixed schema has one right answer in the transcript; variety can only move away from it. That is the same argument the module already makes forenable_thinking, which it disables underjson_modeand leaves alone for prose. Sampling now follows the same boundary.Verified against the real endpoint before writing any code: three identical beat-extraction requests returned three different bodies.
top_k: 1was tried alongside and is redundant — temperature 0 is already greedy here and produced a byte-identical response, so it is not shipped.What it bought, over six runs
Coverage spread falls from 0.636 to 0.182, and its floor rises from 1 of 11 events to 6 of 11. Beat validation spread falls from 0.584 to 0.096. Chronology is now always measurable, where one run previously covered too few events to have an order at all.
The catastrophic draw is gone. The wobble is not.
A correction I need to make
0002fa4's message says "beat validation is now identical across all three runs". True of that sample, and I let it imply determinism. It is not.--repeat 3then gave beat validation 0.750–0.846, so I isolated extraction from compose: three identical extraction calls on the real transcript at temperature 0 returned 26, 26 and 25 beats — two byte-identical, one not.The earlier evidence looked stronger than it was because the determinism probe used a five-line toy prompt, where five identical requests did return five identical bodies. At real scale it does not hold. That is the same shape of mistake this milestone keeps turning up — a check that agrees with the belief that produced it — so it is worth naming rather than quietly restating the numbers.
What is left
Residual nondeterminism has two sources, and neither is sampling.
json_mode=False), deliberately, per #232. Coverage and attribution are scored on the composed prose, which is exactly where the residual spread sits — the fix behaving as designed rather than half-working. But it raises a real question: the beats are already validated when compose runs, so a validated beat that does not reach the prose is information loss, not voice. Whether "did this event survive into the summary" is a stylistic matter at all is worth deciding.--parallel 2with continuous batching: batch composition changes the order of floating-point reductions, so identical greedy requests can diverge. This is a hypothesis, not a measurement — it is not a sampling parameter and no code change here would address it. Testable by running the dev server with--parallel 1and repeating the isolation above.Best-of-n is the promising direction. Beats are validated in code, so running extraction twice and keeping the union of validated beats is cheap, targets coverage directly, and is safe in a way it would not be for prose. Worth trying before chasing bit-determinism, which may not be achievable on a batching server at all.
Criteria
python -m evalsgrows a repeat-count option —--repeat N, with min/mean/max/spread per metric; every run recorded, not just the lastAlso
_structured_openaiand_structured_anthropicare deliberately left at their default temperature, with comments saying so. The o-series rejects any temperature but the default, and Anthropic's extended thinking requires 1 — sending 0 would trade an intermittent quality problem for a hard 400 on exactly the models a GM is most likely to pick, and there is no key here to verify against. #339 was already that mistake once. Belongs with v4.2.0's capability flags; a test pins that both stay bare.Ollama got the change on the same reasoning but was not verified — no instance was reachable. Its
optionsdict is the proven envelope this function already uses fornum_ctx, so it is not a guess at the shape, and a test covers that setting one does not clobber the other. Worth probing on a real Ollama alongside the openformat: <schema>question.Compose work in
899ac1e. The headline result is a negative one, so putting it first: the detector I built to catch dropped events does not fire on real sessions, and the beats-to-prose gap is unchanged at ~0.15.What is demonstrated
The prompt was asking for the drop.
COMPOSE_SYSTEM_PROMPTsaid "cover the key events" — inviting a selection pass over a list that has already been filtered twice, extracted as a beat and then checked against the transcript. There is no third sieve to apply. It now requires every listed event to be represented, while allowing related events to share a sentence and minor ones a clause.The harness was measuring a copy of the pipeline, not the pipeline. This is the find worth keeping. My first version of the compose fix measured as doing nothing at all — and it wasn't nothing:
runner.pyre-implements the compose sequence, so it never executed the new code. A harness built to measure changes to the pipeline could not see a change to the pipeline.recover_dropped_eventsnow takes its compose call as an argument and the harness calls that same function rather than mirroring it. Anything the pipeline does to a summary after composing has to be reachable from there, or the eval quietly stops measuring the product. Worth auditing the rest ofrunner.pyagainst_summarise_from_beatsfor the same class of drift.What is not
unrepresented_beatsflagged nothing across six live runs, while the harness measured 1–2 verified events per run missing from the prose. A ~100% false-negative rate.Two detectors tried, both wrong in opposite directions:
The machinery around it is right and unit-tested — given a summary that drops an event it re-requests it by name and keeps the better draft; given one that recovers nothing it keeps the first. It ships because it is non-fatal, costs nothing when it does not fire, and is correct in the clear-cut cases. But it is not the fix.
Why it is hard, stated plainly: the eval harness only detects these drops because its fixtures carry hand-authored per-event mention groups with curated alternatives. Production has only the beat's own summary text, which is a much weaker signal. Closing that gap needs something other than token overlap — a structured "which of these events did you cover?" check is the obvious candidate, at the cost of one extra call and of being a model judging its own work, which is what this milestone has been trying to get away from.
Numbers, six live runs post-fix
Gap between beats and prose: 0.152, against 0.182 before. Within noise at n=3.
One run did reach coverage 0.909 with
cov.in beats1.000 — so when extraction happens to find everything, compose keeps most of it. That is a hint the extraction ceiling matters as much as the compose leak, which points back at best-of-n extraction as the next thing to try.Also
A test double was returning
"The final summary."for a beat about disarming a rune trap — a compose step that ignored its entire input, i.e. exactly the condition now being checked for. Fixed the double rather than loosening the assertion, per #343: a test double that models an impossible response will eventually assert that the impossible is fine.1,207 passing before, 1,215 after.
Criteria
python -m evalsgrows a repeat-count optionBest-of-2 extraction in
5357022. This is the first change that has moved coverage at all.Measured, three runs each side
Every metric improved. Coverage is up 22% relative, and one run reached 1.000 in beats — all eleven hand-verified events found — where single-pass never exceeded 0.727. The ceiling moved, which neither of the previous two attempts managed.
Two caveats I want on the record. Ranges overlap at n=3, so this is a real signal but not a tight one. And it did not reduce variance the way I predicted when I proposed it: spread went up, not down (coverage 0.182 → 0.273). Averaging more draws improved the mean and evidently not the spread, at least at this sample size. I said best-of-n would steady it as a side effect; that part was wrong.
The beats-to-prose gap is unchanged at ~0.15. Best-of-n raises what compose is handed; compose still discards its usual share. The two problems are independent, exactly as the stage measurement indicated.
Why the union is safe here
Every beat is validated against the transcript by pure code before anything uses it, so taking the union across passes cannot smuggle in a hallucination. The property that makes best-of-n dangerous for prose is the one
validate_beatsalready removes.dedupe_beatsdecides two beats are one event from their cited evidence, not from how alike the summaries read — two passes word the same event differently, so text similarity would be guessing, whereas citing the same transcript line is a fact. Actors must overlap too, since one line can carry two people's actions and shared evidence alone would merge events that merely coincided.Biased towards keeping: a duplicate costs a sentence written twice; an over-merge silently loses an event. A test pins that dedupe is a no-op on a single pass, because an over-eager merge there would drop events for every install that never runs a second one.
The drift, fixed at the cause
The window/extraction loop existed twice — once in the pipeline, once in the harness. That is why the compose fix measured as doing nothing earlier today.
gather_beats_over_windowsis now the single implementation withextractinjected, and the harness calls it. Second time this pattern has bitten, so this fixes the cause rather than the instance. Worth a sweep of anything elserunner.pystill mirrors.Cost, stated rather than buried
This doubles the most expensive call in the pipeline — time on a self-hosted box, money on a metered provider.
BEAT_EXTRACTION_PASSES = 2is a module constant. Whether hosted campaigns should be able to turn it down is a product decision and shouldn't live in a default.1,215 passing before, 1,222 after.
Criteria
python -m evalsgrows a repeat-count optionSuggest the remaining work is: reframe criterion 2 around the floor rather than the spread, and get a detector good enough to satisfy criterion 4 — those are the same problem, since a usable floor check is exactly what would make a bad run visible.
Window sweep done in
8bc9018. The floor is raised: 0.091 when this was filed, 0.727 now.The sweep
Real session, four runs per point, qwen3.5-9B:
Four windows — a ~7,000-token budget, cap
11264— is best or tied-best on everything, and is now the default. Worst-case events reaching the summary: 8 of 11, against 5 of 11 with the whole session in one window.The tail is not what I predicted
I said before running this that going finer would lose events to window boundaries. It doesn't — seven windows holds the same coverage floor. What it loses is attribution (0.804 → 0.615) and validation rate (0.795 → 0.723).
The events are still found; they are attributed worse. Each window carries less surrounding dialogue, so the model has less to ground "who did this" in and cites less reliably. Window size trades finding events against attributing them — and since those are the two failure modes this milestone exists for, the balance point matters more than either alone. That reframes the knob from "how much can the model attend to" to "how much context does an attribution need", which is a different and more useful question.
The dip at 3 windows is unexplained: worse than both neighbours on coverage and attribution, which no smooth story predicts. Most likely where the boundaries happen to fall in this one transcript. Four runs cannot separate that from noise, so it is recorded rather than smoothed away. It does not move the optimum.
The two-way isolation you authorised
It was answerable without GPU time, and the answer was that I had been chasing the wrong variable:
extract_beats' declared context — dead.context_tokensonly ever reaches Ollama'snum_ctx; the llama.cpp path never receives it.fits_in_contextcheck, and_dispatch_prose→ llama.cpp ignores context.extraction_windowsappliesprompt_budget_tokens' 0.7 fraction and the old harness maths did not.So my "clean" run was not a cleaner version of the confounded one; it was a different configuration. I changed the thing under test while removing what I believed was a confound, then read the difference as evidence about compose. Third time this milestone has produced a measurement that agreed with the belief that made it, and the fix was arithmetic rather than another run.
Criteria
Variance reduced to a stated, defensible band→ floor raised, per the reframing: coverage floor 0.091 → 0.455 → 0.727, i.e. worst-case 1 of 11 events to 8 of 11python -m evalsgrows a repeat-count optionWhat remains
Only the GM-visibility criterion, and it still needs a detector I have failed at twice — token overlap against a beat's own summary is too weak a signal, and the harness only manages it because its fixtures carry hand-authored mention groups. Now that the floor is 0.727 rather than 0.091, the case for that detector is weaker than it was; it may be better handled as "the GM can see the verified beat list alongside the summary" than as an automated check.
Suggest closing this and opening a separate issue for that, since it is now a UI question rather than a pipeline one.
Arc of the whole issue: the first three attempts all asked how many times to query. The answer was how much to ask about at once.
Closing. The fourth criterion is handed to #424 in v4.4.0, reframed as showing the GM the verified events rather than detecting a lossy run in code.
What this issue achieved
Coverage floor: 0.091 → 0.727. Worst-case events reaching a GM's summary went from 1 of 11 to 8 of 11.
0002fa4)5357022)8bc9018)What shipped
3bf85b3— beat extraction was silently capped at 2048 output tokens, and failed hardest on the largest context windows. Found by #349's harness on its first real run.0002fa4— no sampling parameters were ever sent, so every structured extraction ran at the server's creative-writing default. Greedy decoding underjson_moderemoved the catastrophic draw.5357022— extraction runs twice and keeps the union of validated beats, merged on cited evidence rather than text similarity.e997926— passes continue while they are still finding events. Did not raise the floor; kept with its negative result recorded.48036ab/8bc9018— extraction windows sized for attention rather than context, swept to an optimum of ~7,000 tokens per window.What I got wrong, since it is the useful part
Three of the four attempts asked how many times to query. The answer was how much to ask about at once.
extraction_windowsappliesprompt_budget_tokens' 0.7 fraction and the old harness maths did not. I changed the thing under test while removing what I believed was a confound.That last one is the pattern worth carrying forward: three separate times this milestone produced a measurement that agreed with the belief that made it. The one time I computed what each configuration actually did before spending GPU on it, arithmetic answered the question outright and saved the run.
Left behind for whoever picks this up
QB_EXTRACTION_CONTEXT_CAP— the window size is a measured value on one model and one session, not a derived one, and will move with the model. Tune it withpython -m evals, not by argument.evals/results/history.jsonl, tagged by configuration.