rbrooks/Quest-Board

Watch

Fork 0

v3.11.2 — the rest of the LLM trigger audit (#271, #285, #286, #287, #288) #290

Merged

claude-bot merged 6 commits from fix/llm-trigger-audit-followups into main

2026-08-06 00:58:42 +00:00

claude-bot commented

2026-08-06 00:46:53 +00:00

Contributor

Completes the LLM trigger audit that produced v3.11.1. Five features wired to the wrong event or to no event at all, plus a broker-blocking fix found while doing them.

Closes #271. Closes #285. Closes #286. Closes #287. Closes #288.

No migrations. BOT_CONTRACT_VERSION stays 1 — nothing touches /api/bot/*.

Commits


`fix(backend)`	budget recent-session context per position (#271)
`fix(backend)`	never leave lore generation stranded mid-flight (#287)
`fix(backend)`	trigger lore summaries and relationships when entries become canon (#285, #286)
`fix(backend)`	record every inline Workbench tool run (#288)
`perf(backend)`	stop request paths and the test suite blocking on an absent broker
`chore(release)`	v3.11.2

Three of these were more than the issue described

#271 was two bugs. Raising the 800-char cap was the obvious fix, but _truncate keeps text[:limit] — the beginning. For a prep sheet the end of the previous summary is the most valuable line in it, which is why the report read as "not pulling end of last session" rather than "truncated". Measured against the live instance, prep was seeing 28-46% of the previous session, always the opening.

#285's most important path changes no data. Approving a bot proposal doesn't touch the body — it only flips proposed_by_bot. Hooking create and update, the obvious reading, would have missed the single most common way entries become canon on a real instance.

#288 had three bypasses, not the two the issue named. The third — the cold-pool draw — is arguably the worst, since a cold draw generates a whole batch and buffers the rest, making it the largest single token spend of the three. It was found by writing the guard test before trusting the issue's list.

The performance change is larger than the regression that caused it

Adding an enqueue to ordinary lore-entry writes (#285) made the suite 4× slower, which exposed something that had always been true: there is no broker in the test environment, so every unmocked .delay() was waiting on a connection that could never succeed.

880 tests: 16m12s → 36s. Most of that "baseline" was never work. CI runs Postgres as a service but no Redis, so it was paying it too.

The same defaults meant a single enqueue against an unreachable broker took ~2 minutes to fail in production — and #285/#286 put enqueues on a user-facing request path (approving a proposal). Bounded to under 15s, with a test asserting that budget.

Deliberately not task_always_eager, which would execute tasks and change test semantics.

Deliberate scope boundaries

#287 fixes the concrete stuck-state bug and the retry-a-permanent-condition problem. The broader "27 call sites, 8 behaviours for LLM-not-configured" harmonisation stays in the issue — it is a refactor with no natural end, and bundling it here would have made this unreviewable.
#286 leaves a known gap tracked in #289: the task resolves its campaign through a session, so an approved entry with no linked_session_id schedules nothing. Making it campaign-scoped is a signature change.
#285 and #286 share a commit — same two functions, same approval path. Splitting them would have meant a third revert-and-reapply cycle on lore_service.py for no revertability gain.

Two stale tests updated rather than deleted

test_lore_consolidate_finds_match_rows_written_by_helper asserted the relationship task was fired at generation time — the exact behaviour #286 removes. It was pinning the bug in place. Now asserts the opposite, with the reasoning inline.

Verification

Backend: 880 passed in 36.36s, full suite, isolated
ruff check + ruff format --check: clean, 194 files
scripts/check_version_sync.py: OK — app 3.11.2, bot contract v1

🤖 Generated with Claude Code

Completes the LLM trigger audit that produced v3.11.1. Five features wired to the wrong event or to no event at all, plus a broker-blocking fix found while doing them. Closes #271. Closes #285. Closes #286. Closes #287. Closes #288. **No migrations.** `BOT_CONTRACT_VERSION` stays 1 — nothing touches `/api/bot/*`. ## Commits | | | |---|---| | `fix(backend)` | budget recent-session context per position (#271) | | `fix(backend)` | never leave lore generation stranded mid-flight (#287) | | `fix(backend)` | trigger lore summaries and relationships when entries become canon (#285, #286) | | `fix(backend)` | record every inline Workbench tool run (#288) | | `perf(backend)` | stop request paths and the test suite blocking on an absent broker | | `chore(release)` | v3.11.2 | ## Three of these were more than the issue described **#271 was two bugs.** Raising the 800-char cap was the obvious fix, but `_truncate` keeps `text[:limit]` — the *beginning*. For a prep sheet the *end* of the previous summary is the most valuable line in it, which is why the report read as "not pulling end of last session" rather than "truncated". Measured against the live instance, prep was seeing 28-46% of the previous session, always the opening. **#285's most important path changes no data.** Approving a bot proposal doesn't touch the body — it only flips `proposed_by_bot`. Hooking `create` and `update`, the obvious reading, would have missed the single most common way entries become canon on a real instance. **#288 had three bypasses, not the two the issue named.** The third — the cold-pool draw — is arguably the worst, since a cold draw generates a whole batch and buffers the rest, making it the largest single token spend of the three. It was found by writing the guard test *before* trusting the issue's list. ## The performance change is larger than the regression that caused it Adding an enqueue to ordinary lore-entry writes (#285) made the suite 4× slower, which exposed something that had always been true: there is no broker in the test environment, so every unmocked `.delay()` was waiting on a connection that could never succeed. **880 tests: 16m12s → 36s.** Most of that "baseline" was never work. CI runs Postgres as a service but no Redis, so it was paying it too. The same defaults meant a single enqueue against an unreachable broker took ~2 minutes to fail *in production* — and #285/#286 put enqueues on a user-facing request path (approving a proposal). Bounded to under 15s, with a test asserting that budget. Deliberately **not** `task_always_eager`, which would execute tasks and change test semantics. ## Deliberate scope boundaries - **#287** fixes the concrete stuck-state bug and the retry-a-permanent-condition problem. The broader "27 call sites, 8 behaviours for LLM-not-configured" harmonisation stays in the issue — it is a refactor with no natural end, and bundling it here would have made this unreviewable. - **#286** leaves a known gap tracked in **#289**: the task resolves its campaign through a session, so an approved entry with no `linked_session_id` schedules nothing. Making it campaign-scoped is a signature change. - **#285 and #286 share a commit** — same two functions, same approval path. Splitting them would have meant a third revert-and-reapply cycle on `lore_service.py` for no revertability gain. ## Two stale tests updated rather than deleted `test_lore_consolidate_finds_match_rows_written_by_helper` asserted the relationship task **was** fired at generation time — the exact behaviour #286 removes. It was pinning the bug in place. Now asserts the opposite, with the reasoning inline. ## Verification - Backend: **880 passed in 36.36s**, full suite, isolated - `ruff check` + `ruff format --check`: clean, 194 files - `scripts/check_version_sync.py`: OK — app 3.11.2, bot contract v1 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 6 commits

2026-08-06 00:46:54 +00:00

fix(backend): budget recent-session context per position (#271 ) 74956e22da

Reported from live use: the GM Workbench session prep sheet "appears to only be
pulling from the first paragraph of last session's summary". It was.

`recent_sessions_context` truncated every recent session to a flat 800 chars.
Real summaries on the live instance run 1725-2874, so the prep sheet was seeing
28-46% of the previous session — always the opening.

Raising the cap alone would have been a half-fix. `_truncate` keeps `text[:limit]`
— the *beginning* — and for a prep sheet the end of the previous summary is the
most valuable line in it: the cliffhanger, the unresolved threat, the door that
just opened. That is why the symptom read as "not pulling end of last session"
rather than merely "truncated".

So the budget is now per position. The most recent session gets 3000 chars,
because "what happened last time" is the entire point of a prep sheet; older
sessions get 1000, which is enough to supply the gist. When the latest summary
does exceed budget, `_truncate_keeping_ends` keeps the opening *and* the closing
with a visible elision marker between them, weighted toward the tail.

Deliberately bounded rather than unlimited: worst case is 3000 + 2x1000 = 5000
chars (~1.2k tokens) of recent-session context. Prompt size is not free —
oversized prompts are what drive this endpoint to emit malformed JSON (#279),
and session_prep is itself a JSON-returning tool. Tripling the cap without a
ceiling would have traded a visible bug for a quieter one.

Affects both consumers of the builder: session_prep (n=2) and arc_suggester
(n=3). Note the codebase already disagreed with itself here — the "previously
on" recap builder caps the same kind of content at 2000.

Adds 10 tests to a builder that had none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

fix(backend): never leave lore generation stranded mid-flight (#287 ) c9ad8fd21c

Each phase of the multi-pass lore pipeline sets `lore_generation_status` to
`extracting` or `matching` before doing work, but nothing cleared it when a task
ran out of retries. Celery raised MaxRetriesExceededError, the task died, and
the session sat in a non-terminal state forever — the UI showing "generating…"
with no error and no way to tell it had stopped.

The trigger is mundane: the GPU box being briefly unreachable during
post-session processing is enough.

Three changes.

`LoreGenerationTask` is a base class whose `on_failure` hook marks the session
failed. Celery fires that once, after the final retry is exhausted, so it runs
exactly when the pipeline has genuinely given up. Attached to all four phases,
with a test asserting that — a fifth phase added later without the base would
otherwise silently reintroduce this.

`LlmNotConfiguredError` separates a permanent condition from a transport
failure. An unconfigured endpoint will still be unconfigured after 30s, 60s and
120s of backoff, so `_retry_unless_permanent` lets it fail straight through to
on_failure instead of burning the retry budget. Retrying a missing setting
turned a clear "configure an LLM" state into a task that died silently several
minutes later.

The failure marker refuses to overwrite a terminal status. The phases fan out,
so a slow match task can fail *after* consolidate has already written `ready`;
marking that session failed would be a lie, and this fix would have introduced
one.

Split into an async core plus a sync wrapper to match the convention every other
task here uses — asyncio.run cannot be called from inside pytest's event loop,
though it is correct in a Celery worker, which has none. on_failure is tested on
its own responsibility (finding the session id and delegating) rather than
through the sync boundary.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

fix(backend): trigger lore summaries and relationships when entries become canon (#285 , #286 ) 15f95e24da

Two mis-wirings from the same audit, in the same code path, so they land
together.

#285 — lore entry summaries were never generated automatically.
`generate_lore_entry_summary` had exactly one enqueue site, the manual
regenerate-summary endpoint, despite a docstring claiming it also fired on entry
creation. Every entry created by any automated path therefore had summary = NULL
permanently, and since bot proposals are how the wiki mostly gets populated
after a session, that was most of the wiki.

The trigger is now "became canon", not "row was written": direct creation,
approval of a bot proposal, and a body edit. The approval hook is the one that
matters and the one easy to miss — approving a proposal changes no body at all,
it only flips proposed_by_bot, so hooking create and update alone would have
missed the most common path. Deliberately not fired when a proposal is first
written, because a proposal may never be approved and summarising rejected ones
spends tokens on content that never becomes canon. A rename or a GM-notes edit
no longer regenerates either — the summary would come out the same.

#286 — relationship proposals fired before the entries they should relate
existed. `propose_lore_relationships` was enqueued the moment the lore pipeline
finished, but it selects entries WHERE proposed_by_bot == False, and everything
the pipeline had just produced was still an unapproved proposal. It ran at the
exact instant new material appeared and structurally could not see any of it, so
it only ever related entries approved *before* the session ran — and re-examined
the same static set every time.

Now scheduled on approval, from the same two branches as the summary trigger,
debounced through a Redis SET NX EX marker: a GM clicking through eight
proposals causes one pass over the final set rather than eight over successive
prefixes.

Both are fire-and-forget with broad exception handling — a summary or a
relationship suggestion is a nice-to-have and must never fail the write that
triggered it. Tested.

Also updates test_lore_consolidate_finds_match_rows_written_by_helper, which
asserted the relationship task WAS fired at generation time. That assertion was
pinning the bug in place, so it now asserts the opposite with the reasoning
inline rather than being deleted.

Known gap, tracked in #289: the task resolves its campaign through a session, so
an approved entry with no linked_session_id schedules nothing. Making it
campaign-scoped is a signature change kept out of this fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

fix(backend): record every inline Workbench tool run (#288 ) 3e783d6b68

`generation_service` is the well-factored part of the LLM layer — one registry,
one executor, one persistence layer in `generation_results` that the Workbench
polls and that token accounting reads. But some endpoints called `run_tool`
directly and never persisted anything, spending real tokens and dropping the
output outside the system of record.

The issue named two. There were three: the campaign name-cache, the legacy
session-scoped name generator, and the cold-pool draw — which is arguably the
worst of them, since a cold draw generates a whole batch, returns one item and
buffers the rest, making it the largest single spend of the three. It was found
by writing the guard test before trusting the issue's list.

Rather than patching three call sites, `generation_result_service.run_and_record`
now owns the create -> run -> mark_ready/mark_failed pattern that was open-coded
in the Workbench endpoint while three other places skipped it. All four go
through it, including the original.

The Redis caches are untouched. They are latency optimisations in front of the
system of record, not replacements for it — a cache hit still short-circuits and
records nothing, which is tested. Failed generations now record a `failed` row
too: a generation that errored still spent tokens trying, so it belongs in
history.

The guard test greps the routers for direct run_tool calls. That is blunt, but
the failure mode it prevents is invisible — a future endpoint bypassing
persistence produces no symptom, just quietly incomplete history and undercounted
spend.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

perf(backend): stop request paths and the test suite blocking on an absent broker 6bc2ef4e2f

Two symptoms of the same thing, found while adding the enqueues in #285/#286.

Celery's defaults let a single `.delay()` against an unreachable broker take
about two minutes to fail — measured, not estimated. Several enqueues run inline
on user-facing request paths (approving a lore proposal, now), so "Redis is
down" presented as an HTTP request that appeared to hang. The try/except around
those enqueues caught the failure; it just caught it two minutes late.

Bounding the socket timeouts and capping task_publish_retry_policy brings the
worst case for a failed enqueue under 15 seconds, with a test asserting that
budget so it cannot silently regress. A second test asserts the 8-day
visibility_timeout still exceeds the 7-day reminder ETA window, since that lives
in the same config block and getting it wrong would cause duplicate reminders.

The test suite was paying the same cost, and had been all along. There is no
broker in the test environment, so every unmocked `.delay()` attempted a real
connection; that was tolerable while enqueues were rare, but #285 puts one on
ordinary lore-entry writes, and hundreds of tests started paying it. Pointing
Celery at kombu's `memory://` transport in conftest makes publishing free.

The effect is larger than the regression that exposed it: the full backend suite
goes from 16m12s to 36s for the same 880 tests. Most of that "baseline" was
never work — it was the suite waiting on a broker that was never there. CI runs
Postgres as a service but no Redis, so it was paying it too.

Deliberately not `task_always_eager`, which would execute tasks and change test
semantics. Tests that assert on enqueues still patch the task and check `.delay`,
unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

chore(release): v3.11.2

CI / Frontend tests, audit, and build (pull_request) Successful in 1m13s

Details

CI / Backend lint (ruff) (pull_request) Successful in 26s

Details

CI / Bot/backend version sync (pull_request) Successful in 21s

Details

CI / Bot tests and audit (pull_request) Successful in 2m40s

Details

CI / Docker image build (pull_request) Successful in 3m0s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 7m24s

Details

60f09c25e7

Patch release completing the LLM trigger audit that produced v3.11.1 — five
features wired to the wrong event or to none at all (#271, #285, #286, #287,
#288), plus the broker-blocking fix found while doing them.

APP_VERSION and BOT_EXPECTED_APP_VERSION move to 3.11.2 together, per the
invariant scripts/check_version_sync.py enforces. BOT_CONTRACT_VERSION stays at
1: nothing in this release touches /api/bot/*, so either image can be upgraded
independently.

No migrations.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>