v4.0.0 — deterministic attribution: the summary is composed from checked events, not asserted #434
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/v4-deterministic-attribution"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes the v4.0.0 milestone — 30 of 30 issues.
The summary is no longer something the model asserts. It is composed from events that have each been checked against the transcript in code, so chronology is a
sorted()and attribution is a lookup rather than a model behaviour.The load-bearing change
#342 removed the unverifiable attribution join. Transcription used to send every speaker's track in one
/transcribe/sessionrequest and recover attribution by mapping each returned label back to the track it was sent for. That join cannot be made safe: a one-directional swap is caught, because the victim track comes back empty, but a symmetric swap is not — every label sent comes back, every segment resolves, every coverage check passes, and the whole session is misattributed with nothing raised and nothing logged.Joining on an id instead of a name would not have closed it either; a permutation of ids survives an id-join exactly as a permutation of names survives a name-join. Any echo-based join is unverifiable in principle, because the only thing that could confirm it is the association being asserted. So there is now one request per track and the label is stamped from the file that was sent. There is nothing left to permute.
If you run with VAD disabled, request volume changes — one request per speaker instead of one per session, to the same endpoint the VAD path already used. VAD is on by default, so most installs are unaffected.
Migrations
Six, applied in order by the
migrateservice:a2b3c4d5e6f8,b3c4d5e6f7a9,c4d5e6f7a8b0,d5e6f7a8b0c1,e6f7a8b0c1d2,f7a8b0c1d2e3. All additive. Onlyc4d5e6f7a8b0backfills — it copies each member's existing character intocampaign_charactersand leaves the old columns in place, so a downgrade is twoDROP TABLEs and no data is at risk.Rehearsed against a full copy of production data before release: all six apply cleanly, row counts unchanged, backfill exact.
Versions
BOT_CONTRACT_VERSIONstays 1, so images can be upgraded one at a time.BOT_EXPECTED_APP_VERSIONmoves to 4.0.0, which only warns on mismatch.Late addition — #431
check_transcript_covers_sessionmeasured the transcript's last timestamp against the recording's wall clock. Every track is tail-padded with silence to that clock and nothing stops a recording when the channel empties, so a table that played for two hours and forgot to stop for another fifty had a complete, correctly attributed transcript rejected for covering "only" 58%.This release made it worse rather than introducing it: the guard predates v4.0.0, but admin retry used to pass
duration=0, which skipped it — so a long-tailed session that failed could at least be recovered. #421'sderive_session_durationarms the guard on exactly that recovery path. Coverage is now measured against how far into the session the tracks carry audible speech.Verification
Known, filed, not blocking
Two defects found while designing the synthetic-audio test harness (#433), both verified in code, both in the v4.0.1 milestone:
speakersis never narrowed afterdrop_silent_tracks, so a backend-dropped silent speaker is still proposed as having spoken. Only reachable on the backend-drop route (an older recording, a hand-placed directory, a bot a version behind) — which is precisely the route #425 exists for.pg_dump17 against a PostgreSQL 16 server and cannot be restored by the host's own tooling. Relevant to anyone taking the CHANGELOG's advice to snapshot before applying these six migrations: take a filesystem or VM snapshot, not just the automatic dump.🤖 Generated with Claude Code
`bot_upload_audio` had no state guard, and `process_audio` unconditionally overwrites transcript and summary and delete-reinserts every highlight. The trigger is not exotic: a GM who stops recording at the break and starts again for the second half submits the same session twice. The second run destroyed the first half's transcript, any edits the GM had made to it, and every curated highlight — and because the bot wrote to a fixed `{session_id}/{user_id}.wav` path, it overwrote the source audio too, so no reprocess could recover it. Three independent holes, closed at three layers: - **Intake refuses.** A session whose `transcript_updated_at` is set now returns 409 rather than queueing. That field is the right marker because it is set whenever the transcript is written *or edited*, so a hand-edited transcript is protected as firmly as a generated one. `force=true` replaces deliberately. - **Highlights survive.** The delete before reinsert was by `session_id` alone. It is now scoped to rows that are neither approved nor GM-edited, so curation — including manually added highlights — survives a reprocess. - **Takes no longer collide.** The bot writes a second take to `{session_id}-take2/` instead of over the first. The first take keeps the bare session id, so nothing about the backend contract or the handoff sweep changes. The bot distinguishes the refusal from an outage. A 409 raises `AudioAlreadySubmittedError`, and the GM is told their existing transcript was kept and where this take is saved — rather than "could not be submitted… an admin can retry", which would be both wrong and alarming for what is actually the system protecting their work. The bot never sends `force`, so it cannot destroy a transcript at all; that requires a deliberate admin action. Deliberately **not** bumping BOT_CONTRACT_VERSION. The request change is additive, and while a previously-202 case can now return 409, an older bot handles that by telling the GM it could not submit — which is true, harmless, and the safest possible degradation. Forcing every self-hoster into a lockstep image upgrade for a slightly wrong error message in a rare case is disproportionate; both images move together in v4.0.0 regardless. This was moved into v4.0.0 from v4.1.0 because it sits in the path of the accuracy work: developing the summarisation re-architecture means re-running `process_audio` against real sessions repeatedly, and until now every one of those runs was destructive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Member erasure found a speaker's lines by matching the rendered transcript against their character_name or display_name. Neither is necessarily the label they were rendered under: the label is resolved per session from their guild nickname, and gets a suffix when it collides with someone else's. So an erasure request could complete, report success, and leave the member's words in place. The same lookup decided which quote highlights to delete, with the same blind spot, and the same collision could cut the other way — two members rendering under one label meant erasing either destroyed both. Transcript rows carry track_owner_id, the Discord account whose audio file the words came from, so erasure now keys off that. Sessions recorded before rows existed keep the label path; it is never backfilled, so there is nothing better to match on there. Two things the migration must not break, both pinned by tests: * A GM's hand-edits are never re-rendered over. We re-render from rows only when the stored text is still byte-for-byte the render of those rows. Once it has diverged we scrub the text the GM has — using the label learned from their rows, so the erasure is still complete. * A member with no verified Discord link has no authoritative key, so the label path still runs for them rather than being skipped because rows exist. Where the member has a link but a session's lines carry their name under a different track, we log rather than guess: matching the label anyway could destroy a different member's words, and an account change is worth a human look. Also closes a hole #335 opened: the retention sweep cleared sessions.transcript but left the rows holding the same words, so an expired transcript was still sitting in the place the pipeline reads from. The line format now has exactly one definition, render_transcript_line, which was the point of moving structure into rows — beat_service parses that shape back out to verify cited evidence, and it should not be spelled out twice. Also reformats five test files added earlier on this branch: CI runs `ruff format --check` over webapp/backend, and only `ruff check` had been run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>JSON output on the Anthropic path was steered by prefilling the assistant turn with "{". That was removed on Claude 4.6 and the entire 5-family and now returns a 400 — so an operator who configured any current model broke every structured feature in the product at once, with an error reading like a Quest Board bug. The pinned default snapshot still accepted it, which is exactly why this sat latent. The replacement is a schema rather than a nudge, and it goes further than the Anthropic path: all four providers support constrained output, so generate_structured_text now takes an optional json_schema and each provider spells it its own way. Two of those spellings look interchangeable and are not — llama.cpp puts the schema directly under response_format, OpenAI nests it under a further json_schema key — which is the same kind of runtime-only, wrong-backend failure the prefill was. They live in one function, with a test per provider pinning the exact wire shape. On llama.cpp and Ollama a schema becomes a decoding grammar, so a malformed response is impossible rather than merely discouraged. That matters more than the Anthropic fix: it is what lets the accuracy-critical paths run on a small self-hosted model instead of degrading to the unverified prose fallback every time a 7B model fumbles a brace. Beat extraction and quote highlights — the two paths the v4 accuracy work rests on — now declare their shape. The other callers have no fixed shape and keep the tolerant parser, unchanged. Both shipped schemas are checked by a test against the strictest provider's rules (additionalProperties false everywhere, every property required), since violating either is rejected at request time on a provider the developer may not be testing against. Documents provider selection, the model field, and the context-window setting in OPERATIONS.md, including the note that dated Anthropic snapshots and current ids do not behave identically. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Coverage was only ever measured on the finished prose, so a missing event told you it was missing and nothing about where it went. There are two stages that can lose it and they need opposite fixes: an event that was never extracted cannot be recovered by tuning compose, and one that was extracted, validated and then left out of the prose will not be recovered by extracting harder. `coverage_in_beats` scores the same ground-truth events against the verified beats, before compose runs. Free — `score_coverage` is pure code and the rendering already exists — so it is always measured rather than hidden behind a flag, and it appears in both the per-run table and the `--repeat` variance table. Measured on the real session, three runs, per event: coverage in verified beats 0.727 0.818 0.636 (mean 0.727) coverage in composed prose 0.455 0.636 0.545 (mean 0.545) never extracted 1/11 lost or flaky at compose 3/11 survived both stages 7/11 So compose discards roughly a quarter of what extraction successfully found and the validator confirmed. One event — Wyatt finding the corridor of amphora — was extracted and verified in all three runs and dropped from the prose in all three. That is not sampling noise; it is a consistent editorial choice. Which settles the question that prompted this measurement. Making compose greedy would have made that drop *reliable* rather than fixed it, so the compose problem is not a sampling problem: it is that compose may silently discard an event the validator has already confirmed happened. Worth noting the fixture weights all eleven events equally while a summary legitimately compresses — but the human-approved reference summary mentions all three dropped events, so the reference disagrees with compose in every case. 1,207 passing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Extraction is not deterministic even at temperature 0, and the draws are not nested: three identical calls on one real session returned 26, 26 and 25 beats, and the 25 was not a subset of the 26. Different passes surface different events. Since every beat is validated against the transcript by pure code before anything uses it, taking the union across passes cannot smuggle in a hallucination — the property that makes best-of-n dangerous for prose is exactly the one the validator already removes here. ## Measured, three runs each metric single pass best of two coverage (prose) 0.545 (0.455-0.636) 0.667 (0.545-0.818) coverage in beats 0.697 (0.636-0.727) 0.818 (0.636-1.000) attribution 0.510 (0.429-0.600) 0.542 (0.500-0.571) chronology (tau) 0.794 (0.714-0.867) 0.868 (0.833-0.905) beat validation 0.740 (0.625-0.846) 0.812 (0.714-0.923) Every metric improved. Coverage is up 22% relative, and one run reached 1.000 in beats — all eleven hand-verified events found — where single-pass never exceeded 0.727. The ceiling moved, which is the thing the last two attempts could not do. Two honest caveats. Ranges overlap at n=3, so this is a real signal rather than a tight one. And it did **not** reduce variance the way I expected: spread went up, not down (coverage 0.182 to 0.273). Averaging more draws makes the mean better and evidently not the spread, at least at this sample size. The beats-to-prose gap is unchanged at ~0.15. Best-of-n raises what compose is handed; compose still discards its usual share. The two problems are independent, as the stage measurement said they were. ## How the union is merged `dedupe_beats` decides two beats are one event from their **cited evidence**, not from how alike the summaries read. Two passes word the same event differently, so text similarity would be a guess; citing the same transcript line is a fact, and keeping the merge grounded in the transcript is the principle the validator already runs on. Actors must overlap too — a single line can carry two people's actions, so shared evidence alone would merge events that only happened at the same moment. Biased towards keeping. A duplicate costs a sentence written twice, which a reader notices and shrugs at; an over-merge silently loses an event, which is the failure this milestone exists to stop. There is a test that dedupe is a no-op on a single pass, because an over-eager merge there would drop events for every install that never runs a second one. Validate first, then dedupe: the other order would let a beat that cannot pass validation displace the pass that got the same event right. ## And the drift, fixed at the cause The window/extraction loop existed twice — once in the pipeline, once in the eval harness. That duplication is why a compose fix measured as doing nothing an hour ago: the harness never ran it. `gather_beats_over_windows` is now the one implementation, with `extract` injected, and the harness calls it. Second time this pattern has bitten, so this fixes the cause rather than the instance. ## Cost This doubles the most expensive call in the pipeline — time on a self-hosted box, money on a metered provider. `BEAT_EXTRACTION_PASSES` is a module constant; whether hosted campaigns should be able to turn it down is a product decision, not one to bury in a default. 1,215 passing before, 1,222 after. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>An attempt at raising the floor that did not raise the floor. Recording it with its numbers rather than quietly reverting, because the reason it failed is worth more than the change. Fixed two passes raised the mean and left the worst run alone, which makes sense: the worst run is the one where both draws were poor, and no fixed count rescues it. So passes now continue while they are still paying and stop on the first that adds nothing new, bounded 2 to 4. A session whose passes agree costs what it cost before; only sessions going badly buy more draws. ## It did not work, and the premise is why metric fixed best-of-2 (n=3) adaptive (n=4) coverage 0.667 floor 0.545 0.568 floor 0.455 coverage in beats 0.818 floor 0.636 0.750 floor 0.636 The coverage-in-beats floor is *identical*. Not improved, not noisily similar — the same number. **Agreement between passes is not evidence of completeness, and the stopping rule assumed it was.** A bad run is not one where passes keep turning up new events; it is one where the model consistently misses the same events. Pass two agrees with pass one's poor view, adds nothing, the loop stops — precisely when continuing would have helped. The extra passes never fire on the runs that need them, which is why the floor cannot move. The mean also came out lower than fixed-2, though at n=3 and n=4 with heavily overlapping ranges I would not read that as a real regression. The floor number is the one that means something, and it did not move. ## Why it ships anyway With MIN=2 this is fixed-2 plus passes that only run when they are productive, so it cannot extract less than before. It is more code for no measured gain, which I would normally revert — but it does strictly more work exactly when there is more to find, and the failure is in what the stopping rule *proves*, not in what the loop does. Validation happens per pass before merging, so "found something new" means something new that survives checking. Otherwise a model inventing fresh events every pass would look like progress and run to the cap for nothing. There is a test for that, and one that a session whose passes agree still costs the minimum. A test of my own caught something about the deduper on the way: the first version gave every synthetic "distinct" event the same timestamp and actor, and the loop stopped at two. That was correct — those are one event by the merge rule, since they cite the same line. The fixture was wrong, not the code. ## What to try next The floor run finds 7 of 11 events. Extraction currently gets the whole 77-minute session as a single window, because window size is derived from the provider's context and this model has 131k of it. That is a lot of transcript to ask for uniform attention across, and #340/#341 optimised for fitting *more* in — which may be the wrong direction for extraction specifically. Smaller windows are testable today with `--context-tokens`, no code change needed. 1,222 passing before, 1,225 after. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Swept extraction window size on the real session, four runs per point, against qwen3.5-9B: windows coverage floor coverage mean attribution beat validation 1 0.455 0.568 0.533 0.747 2 0.545 0.614 0.810 0.729 3 0.455 0.523 0.425 0.736 4 0.727 0.818 0.804 0.795 7 0.727 0.773 0.615 0.723 Four windows — a ~7,000-token budget, cap 11264 — is best or tied-best on every metric, so that is the default. The coverage floor goes from 5 of 11 events to 8 of 11 worst-case, against 5 of 11 when the whole session went in one window. ## The tail says something I did not predict Going finer than four does not cost coverage: seven windows holds the same floor, 0.727. What it costs is **attribution** — 0.804 down to 0.615 — and validation rate, 0.795 to 0.723. That is not the boundary-loss effect I predicted before running this. The events are still found; they are attributed worse, because each window carries less surrounding dialogue for the model to ground "who did this" in and to cite from. Window size trades finding events against attributing them, and those two are what this milestone is actually about, so the balance point matters more than either alone. The dip at three windows is unexplained. It is worse than both its neighbours on coverage and attribution, which no smooth story predicts; most likely it is where the boundaries happen to fall in this one transcript, and four runs cannot separate that from noise. Recorded rather than smoothed over — it does not move the optimum. ## Where #423 stands Coverage floor across the whole issue: 0.091 when it was filed, 0.455 after greedy decoding removed the catastrophic draw, 0.727 now. The thing that mattered was not how many times to ask, which is what the first three attempts all tried, but how much to ask about at once. `QB_EXTRACTION_CONTEXT_CAP` keeps it tunable. This is one model on one session, and the useful size will move with the model — the point of #349 is that the next person can measure that rather than argue about it. 1,227 passing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>`drop_silent_tracks` runs on every session and drops any track whose audio never rises above the floor. It decided that by probing 32 fixed positions, reading at most one second at each — while the *spacing* between probes scaled with file length. On a 77-minute session that is 32 seconds inspected out of 4,639, with 145-second gaps, so the guard only ever guaranteed finding a speaker whose audio held one contiguous block longer than that gap. A quiet player does not speak in one block. They say a few words every several minutes, which is exactly the distribution that falls between probes — and the consequence is silent: the track is dropped before transcription, the player vanishes from the transcript, and the pipeline reports success. Measured against the old implementation, whether someone survived came down to whether their speech happened to line up with the probe positions: 40 utterances x 4s (163s of speech) kept 20 utterances x 4s (80s of speech) DROPPED 10 utterances x 3s (30s of speech) DROPPED 5 utterances x 2s (10s of speech) kept 3 utterances x 2s (6s of speech) kept 8 utterances x 1s (8s of speech) DROPPED one 0.5s word in an hour DROPPED Eighty seconds of speech deleted while six seconds survives. Not a threshold anyone chose — an alignment artefact between two evenly-spaced sequences. `has_audible_speech` reads sequentially and stops at the first sample loud enough. Exact, no sampling, and *faster* in the case that matters: anyone who speaks ends the loop as soon as they do, and only a genuinely silent track is read to the end — which is the one file worth being certain about. ## The old test could not have caught this `test_one_brief_utterance_is_enough_to_keep_a_track` cites this exact incident in its docstring — "Sol_Invictus spoke for 163 seconds out of 4639" — and then builds a 60-second file. At that length the probes are 1.9s apart and 1s wide, so coverage is 53% and the assertion cannot fail whatever the code does. Green, and validating nothing. I nearly repeated it. My first replacement used 40 utterances of 4 seconds at real session length, which *looks* like the harder case — and the old code keeps it, because evenly-spaced speech resonates with evenly-spaced probes. I checked by rebuilding the old implementation and running the new fixtures against it, rather than assuming a longer file was automatically a better test. The shapes that ship are ones measured to fail before the fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>