[Backend] Validate highlights' speaker and timestamp against real segments #346

New issue

Closed

opened 2026-08-25 20:39:03 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-08-25 20:39:03 +00:00

Contributor

Severity: MEDIUM. Found in the August 2026 session lifecycle review (#319).

The defect

extract_highlights does the hard half right — every quote is validated by normalised substring match against the transcript, so a hallucinated quote is dropped rather than published. But the speaker and timestamp_ref stored on each SessionHighlight row are LLM-asserted strings that are never checked (reminder_tasks.py:2252-2261).

So a quote can be genuine, verbatim, and attributed to the wrong player — and these are published to players on the Quote Board, which makes a misattribution both visible and socially awkward in a way a summary error is not.

Proposed fix

Once segment rows exist, validate both fields against them: the timestamp_ref must match a real segment, and the speaker must be that segment's resolved owner. On mismatch, correct the speaker from the segment (the segment is authoritative) rather than dropping an otherwise good quote, and log the correction rate as a model-quality signal.

Acceptance criteria

timestamp_ref is validated against a real segment
speaker is taken from the segment, not from the model
Corrections are counted and logged
Tests cover a correct quote, a right-quote-wrong-speaker case, and an invented timestamp

**Severity: MEDIUM.** Found in the August 2026 session lifecycle review (#319). ## The defect `extract_highlights` does the hard half right — every quote is validated by normalised substring match against the transcript, so a hallucinated quote is dropped rather than published. But the `speaker` and `timestamp_ref` stored on each `SessionHighlight` row are **LLM-asserted strings that are never checked** (`reminder_tasks.py:2252-2261`). So a quote can be genuine, verbatim, and attributed to the wrong player — and these are published to players on the Quote Board, which makes a misattribution both visible and socially awkward in a way a summary error is not. ## Proposed fix Once segment rows exist, validate both fields against them: the `timestamp_ref` must match a real segment, and the `speaker` must be that segment's resolved owner. On mismatch, correct the speaker from the segment (the segment is authoritative) rather than dropping an otherwise good quote, and log the correction rate as a model-quality signal. ## Acceptance criteria - [ ] `timestamp_ref` is validated against a real segment - [ ] `speaker` is taken from the segment, not from the model - [ ] Corrections are counted and logged - [ ] Tests cover a correct quote, a right-quote-wrong-speaker case, and an invented timestamp

claude-bot added this to the v4.0.0 - Session Pipeline Accuracy milestone

2026-08-25 20:39:03 +00:00

claude-bot added the

bug

backend

labels

2026-08-25 20:39:03 +00:00

rbrooks referenced this issue from a commit

2026-08-26 02:09:26 +00:00

fix(backend): take a quote's speaker from the transcript, not the model (#346)

claude-bot commented

2026-08-27 23:27:32 +00:00

Author

Contributor

Verified against the acceptance criteria before closing. All met.

anchor_highlights (audio_service.py:2654-2731) matches each quote against the real segment list by normalised substring, then takes both the timestamp and the speaker from the matched segment, overriding whatever the model claimed (true_speaker = chosen.get("speaker"), line 2716). Corrections are counted in a HighlightAnchoring record (speaker_corrected / timestamp_corrected / ambiguous / dropped) and logged (reminder_tasks.py:2332-2337).

All three named test cases exist explicitly: test_a_correct_quote_is_left_alone, test_a_real_quote_credited_to_the_wrong_player_is_corrected, test_an_invented_timestamp_is_replaced_by_the_real_one (tests/test_highlight_anchoring.py:49-84), plus an ambiguous-match case and a cross-speaker stitched quote being dropped.

Wired into the live pipeline at reminder_tasks.py:2299-2331 against all_segments — the same per-track ASR output persisted as TranscriptSegment rows a few lines above, so the anchor and the rows cannot disagree.

Closing. Part of a full acceptance-criteria pass across the v4.0.0 milestone.

Verified against the acceptance criteria before closing. All met. `anchor_highlights` (`audio_service.py:2654-2731`) matches each quote against the real segment list by normalised substring, then takes both the timestamp and the speaker from the **matched segment**, overriding whatever the model claimed (`true_speaker = chosen.get("speaker")`, line 2716). Corrections are counted in a `HighlightAnchoring` record (`speaker_corrected` / `timestamp_corrected` / `ambiguous` / `dropped`) and logged (`reminder_tasks.py:2332-2337`). All three named test cases exist explicitly: `test_a_correct_quote_is_left_alone`, `test_a_real_quote_credited_to_the_wrong_player_is_corrected`, `test_an_invented_timestamp_is_replaced_by_the_real_one` (`tests/test_highlight_anchoring.py:49-84`), plus an ambiguous-match case and a cross-speaker stitched quote being dropped. Wired into the live pipeline at `reminder_tasks.py:2299-2331` against `all_segments` — the same per-track ASR output persisted as `TranscriptSegment` rows a few lines above, so the anchor and the rows cannot disagree. Closing. Part of a full acceptance-criteria pass across the v4.0.0 milestone.

claude-bot closed this issue

2026-08-27 23:27:51 +00:00

rbrooks referenced this issue from a commit

2026-09-05 04:31:28 +00:00

feat(audio): anchor highlights to words