rbrooks/Quest-Board

Watch

Fork 0

[Backend] Verify the returned speaker set matches the tracks that were sent #343

New issue

Closed

opened 2026-08-25 20:39:01 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-08-25 20:39:01 +00:00

Collaborator

Severity: MEDIUM. Found in the August 2026 session lifecycle review (#319).

The defect

The backend's failure handling is otherwise good — a non-200 raises, exceptions mark the session failed with the error stored and the bot notified, and no partial transcript is persisted. But there is a gap on the default path: the remote server can skip an unreadable or failed track and return 200 with the remaining speakers' segments. The backend filters empty-text segments silently (audio_service.py:539) and never checks that every speaker it sent produced at least one segment.

A whole missing player is exactly "misattributed actions" from the GM's point of view — that player's deeds survive only through other people's reactions to them, so the summary credits whoever reacted.

Proposed fix

After transcription, assert that the set of speakers represented in the returned segments equals the set of tracks sent. On a mismatch, fail into the existing failed-status path naming the missing speaker.

Pair it with a softer signal: compare each track's segment count against its speech duration and warn when a track with substantial audio produced few or no segments. Once segment rows land, this becomes a query rather than a special case.

Acceptance criteria

Returned speaker set is compared against the sent set; a mismatch fails loudly
The error names the missing speaker(s)
A track with speech but no segments produces a warning
Tests cover a response omitting one speaker entirely

**Severity: MEDIUM.** Found in the August 2026 session lifecycle review (#319). ## The defect The backend's failure handling is otherwise good — a non-200 raises, exceptions mark the session failed with the error stored and the bot notified, and no partial transcript is persisted. But there is a gap on the default path: the remote server can **skip an unreadable or failed track and return 200** with the remaining speakers' segments. The backend filters empty-text segments silently (`audio_service.py:539`) and never checks that every speaker it sent produced at least one segment. A whole missing player is exactly "misattributed actions" from the GM's point of view — that player's deeds survive only through other people's reactions to them, so the summary credits whoever reacted. ## Proposed fix After transcription, assert that the set of speakers represented in the returned segments equals the set of tracks sent. On a mismatch, fail into the existing failed-status path naming the missing speaker. Pair it with a softer signal: compare each track's segment count against its speech duration and warn when a track with substantial audio produced few or no segments. Once segment rows land, this becomes a query rather than a special case. ## Acceptance criteria - [ ] Returned speaker set is compared against the sent set; a mismatch fails loudly - [ ] The error names the missing speaker(s) - [ ] A track with speech but no segments produces a warning - [ ] Tests cover a response omitting one speaker entirely

claude-bot added this to the v4.0.0 - Session Pipeline Accuracy milestone

2026-08-25 20:39:01 +00:00

claude-bot added the

bug

backend

recording

labels

2026-08-25 20:39:01 +00:00

claude-bot commented

2026-08-26 00:12:42 +00:00

Author

Collaborator

Partly addressed on feat/v4-deterministic-attribution, with a correction to this issue's premise.

The check as written — "assert the returned speaker set equals the sent speaker set" — would produce false failures. A participant who joins voice and never speaks legitimately yields zero segments, and with silence trimming enabled their track is trimmed to nothing, so an equality assertion would fail a perfectly good session.

What landed instead, split by what each case actually means:

A returned label that was never sent now fails loudly. That is the server inventing, renaming or dropping a speaker, and attribution genuinely cannot be trusted. Previously it fell through to the literal string "Unknown", which then flowed into the transcript as though it were a person.
A sent track that produced no segments now logs a warning naming the speakers. Not an error, because silence is a legitimate outcome — but a track that should have had speech and produced none is exactly the shape of a silently dropped speaker, and it was previously invisible.

What remains open here is the stronger version: comparing each track's segment count against its measured speech duration, so "this track had 40 minutes of speech and produced nothing" is distinguishable from "this person was quiet". That needs the segment rows from #335 to be a query rather than a special case, so it is better done there.

Partly addressed on `feat/v4-deterministic-attribution`, with a correction to this issue's premise. The check as written — "assert the returned speaker set equals the sent speaker set" — would produce false failures. A participant who joins voice and never speaks legitimately yields zero segments, and with silence trimming enabled their track is trimmed to nothing, so an equality assertion would fail a perfectly good session. What landed instead, split by what each case actually means: - **A returned label that was never sent now fails loudly.** That is the server inventing, renaming or dropping a speaker, and attribution genuinely cannot be trusted. Previously it fell through to the literal string `"Unknown"`, which then flowed into the transcript as though it were a person. - **A sent track that produced no segments now logs a warning naming the speakers.** Not an error, because silence is a legitimate outcome — but a track that *should* have had speech and produced none is exactly the shape of a silently dropped speaker, and it was previously invisible. What remains open here is the stronger version: comparing each track's segment count against its measured speech duration, so "this track had 40 minutes of speech and produced nothing" is distinguishable from "this person was quiet". That needs the segment rows from #335 to be a query rather than a special case, so it is better done there.

rbrooks referenced this issue from a commit

2026-08-26 15:00:30 +00:00

fix(backend): fail when a speaker we sent comes back with nothing (#343)

rbrooks referenced this issue from a commit

2026-08-26 20:15:32 +00:00

fix(backend): stop compose silently discarding verified events (#423)

claude-bot referenced this issue

2026-08-26 20:15:53 +00:00

[Backend] Summary quality varies 8x run-to-run on identical input #423

claude-bot commented

2026-08-27 23:28:05 +00:00

Author

Collaborator

Verified before closing, against the criteria as amended by the 2026-08-26 comment — the amendment is the right yardstick, and everything it claims landed is genuinely there.

Unrecognised label fails loudly — audio_service.py:629-651, RuntimeError for any label never sent.
Missing speaker is an error, naming who and how much — check_every_speaker_was_transcribed (audio_service.py:1274-1302); test_the_error_names_who_is_missing_and_how_much_they_said asserts both the name and the seconds appear.
A barely-speaking player is warned about, not failed — gated on SPEAKER_MISSING_MIN_SPEECH_SECONDS = 30.0. Tested by test_a_barely_speaking_player_is_warned_about_not_failed.

The amendment itself was the right call: strict set equality would have false-failed a legitimate non-speaking participant, which is the quiet-player mistake in a new place.

Two notes for the record.

The warning-not-error behaviour applies only on the VAD path, where per-track speech duration is known. On the plain transcribe_session path there is no duration signal, so any absence is an error — test_the_non_vad_path_treats_any_absence_as_a_failure pins that. Correct, but materially narrower than "a track with speech but no segments produces a warning" reads as a blanket rule.

The stronger check the comment deferred — comparing segment count against measured speech duration — is still not implemented, and #335 did not bring it either. It is not tracked anywhere as its own issue, so closing this without noting it would make it invisible. Worth filing if it still matters; the segment rows it was waiting on now exist.

Closing. Part of a full acceptance-criteria pass across the v4.0.0 milestone.

Verified before closing, against the criteria **as amended by the 2026-08-26 comment** — the amendment is the right yardstick, and everything it claims landed is genuinely there. - **Unrecognised label fails loudly** — `audio_service.py:629-651`, `RuntimeError` for any label never sent. - **Missing speaker is an error, naming who and how much** — `check_every_speaker_was_transcribed` (`audio_service.py:1274-1302`); `test_the_error_names_who_is_missing_and_how_much_they_said` asserts both the name and the seconds appear. - **A barely-speaking player is warned about, not failed** — gated on `SPEAKER_MISSING_MIN_SPEECH_SECONDS = 30.0`. Tested by `test_a_barely_speaking_player_is_warned_about_not_failed`. The amendment itself was the right call: strict set equality would have false-failed a legitimate non-speaking participant, which is the quiet-player mistake in a new place. **Two notes for the record.** The warning-not-error behaviour applies only on the **VAD path**, where per-track speech duration is known. On the plain `transcribe_session` path there is no duration signal, so *any* absence is an error — `test_the_non_vad_path_treats_any_absence_as_a_failure` pins that. Correct, but materially narrower than "a track with speech but no segments produces a warning" reads as a blanket rule. The stronger check the comment deferred — comparing segment count against measured speech duration — is **still not implemented**, and #335 did not bring it either. It is not tracked anywhere as its own issue, so closing this without noting it would make it invisible. Worth filing if it still matters; the segment rows it was waiting on now exist. Closing. Part of a full acceptance-criteria pass across the v4.0.0 milestone.

claude-bot closed this issue

2026-08-27 23:28:35 +00:00