rbrooks/Quest-Board

Watch

Fork 0

[Backend] Compact the transcript prompt — merge same-speaker runs and use block timestamps #341

New issue

Closed

opened 2026-08-25 20:38:59 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-08-25 20:38:59 +00:00

Contributor

Severity: MEDIUM. Found in the August 2026 session lifecycle review (#319). Cheap, architecture-independent, and worth doing regardless of which summarisation design lands.

The opportunity

Roughly 35% of a transcript prompt is per-line overhead. The format [HH:MM:SS] Name: costs 10-14 tokens per line: about 7 for the bracketed stamp, 2-5 for the name, one newline. At ~1,700 lines centrally, that is ~20k tokens of pure tax out of a ~55k prompt.

Proposed fix

Two changes with no information loss that matters to a summary:

Run-length merge consecutive same-speaker segments into one line. In practice a speaker's utterances cluster, so this removes a large fraction of the labels.
Coarsen timestamps to per-block headers (e.g. one stamp per 30 s block) rather than one per line, keeping exact per-line stamps only where the beat pipeline needs citations.

Together these recover an estimated 10-15k tokens on every full-transcript prompt. Measure before and after on a real transcript rather than trusting the estimate.

Note the coupling: any format change must update _TRANSCRIPT_LINE_RE (reminder_tasks.py:2786), the highlights prompt (audio_service.py:933-951), and every prompt that describes the line format. Landing the segment-rows issue first makes this much safer, since the text becomes a render rather than the source of truth.

Acceptance criteria

Consecutive same-speaker segments are merged in the rendered transcript
Timestamps are emitted per block, with per-line precision retained where citations require it
Token reduction measured on a real transcript and recorded in the issue
All format consumers updated in lockstep
Beat evidence references still resolve after the format change

**Severity: MEDIUM.** Found in the August 2026 session lifecycle review (#319). Cheap, architecture-independent, and worth doing regardless of which summarisation design lands. ## The opportunity Roughly **35% of a transcript prompt is per-line overhead**. The format `[HH:MM:SS] Name: ` costs 10-14 tokens per line: about 7 for the bracketed stamp, 2-5 for the name, one newline. At ~1,700 lines centrally, that is ~20k tokens of pure tax out of a ~55k prompt. ## Proposed fix Two changes with no information loss that matters to a summary: 1. **Run-length merge consecutive same-speaker segments** into one line. In practice a speaker's utterances cluster, so this removes a large fraction of the labels. 2. **Coarsen timestamps to per-block headers** (e.g. one stamp per 30 s block) rather than one per line, keeping exact per-line stamps only where the beat pipeline needs citations. Together these recover an estimated 10-15k tokens on every full-transcript prompt. Measure before and after on a real transcript rather than trusting the estimate. Note the coupling: any format change must update `_TRANSCRIPT_LINE_RE` (`reminder_tasks.py:2786`), the highlights prompt (`audio_service.py:933-951`), and every prompt that describes the line format. Landing the segment-rows issue first makes this much safer, since the text becomes a render rather than the source of truth. ## Acceptance criteria - [ ] Consecutive same-speaker segments are merged in the rendered transcript - [ ] Timestamps are emitted per block, with per-line precision retained where citations require it - [ ] Token reduction measured on a real transcript and recorded in the issue - [ ] All format consumers updated in lockstep - [ ] Beat evidence references still resolve after the format change

claude-bot added this to the v4.0.0 - Session Pipeline Accuracy milestone

2026-08-25 20:38:59 +00:00

claude-bot added the

enhancement

backend

labels

2026-08-25 20:38:59 +00:00

rbrooks referenced this issue from a commit

2026-08-26 02:31:00 +00:00

perf(backend): merge same-speaker runs before prompting (#341)

claude-bot commented

2026-08-26 02:31:20 +00:00

Author

Contributor

Measured on real transcripts

Run against all seven production transcripts over 20k chars, in the prod backend container (read-only; no transcript text left the container). chars/token = 3.4, the codebase's own estimate.

session	lines	→	chars	→	saved	tokens
0254b165	2522	1834	143,340	126,049	12.1%	~5,085
56cc4dee	1904	1305	114,283	99,079	13.3%	~4,471
a98d9eb8	1372	1042	93,409	85,012	9.0%	~2,469
b7c7ae47	1407	1030	90,362	81,415	9.9%	~2,631
d633ad3c	1282	902	81,302	71,724	11.8%	~2,817
09168e79	1212	903	79,726	72,271	9.4%	~2,192
79b7b2d0	1107	838	70,197	63,599	9.4%	~1,940
total			672,619	599,149	10.9%	~21,608

The estimate was optimistic. The issue projected ~35% of the prompt as per-line overhead and 10-15k tokens recovered. Actual: 10.9% of characters, ~1,900-5,100 tokens per session. Line counts fell 24-30%, not "a large fraction" — speech clusters, but less than assumed. Still worth doing: it is free, lossless, and it is the margin that keeps a borderline session out of the chunked path.

Deviations from the proposed fix

1. Runs are capped at 30 seconds. Merging without a time bound collapses a five-minute GM monologue to one line at one timestamp — every beat drawn from it then shares a start time, so ordering within the monologue is lost and cited evidence can sit minutes from what it describes. Three existing beat tests failed on exactly that shape, which is how it surfaced. The cap costs ~1 percentage point (11.9% → 10.9%).

2. Per-block timestamps are NOT implemented, deliberately. Under block stamps a citation can only say "somewhere in this 30 seconds", and a block holds several speakers — so "this actor spoke this line" degrades to "this actor spoke near this line". That is precisely the verification #332/#333/#334 exist to provide, and it is worth more than the remaining tokens. The issue's own hedge — "keeping exact per-line stamps only where the beat pipeline needs citations" — resolves to everywhere, because any line can be evidence.

If the token pressure ever justifies revisiting this, the safe version is block headers in the prompt plus a per-line stamp retained on every line, which saves nothing — so realistically this half of the issue is closed as not-worth-doing rather than deferred.

Consumers

The format did not change, so _TRANSCRIPT_LINE_RE, the highlights prompt and the beat prompts needed no edit. Compaction applies to the prompt only — session.transcript is untouched, and every consumer downstream of summarise() reads the same compacted text, since the validator resolves stamps against it and indexing a different rendering than the model saw would fail every citation.

beat_service._LINE_RE is now public LINE_RE — it is the shared definition of the line format, and re-declaring it per consumer is the coupling #335 set out to remove.

Acceptance criteria: merged ✓, block timestamps ✗ (closed as harmful, reasoning above), measured ✓, consumers unchanged ✓, beat references still resolve ✓ (test_every_remaining_stamp_still_indexes_to_a_real_line).

## Measured on real transcripts Run against all seven production transcripts over 20k chars, in the prod backend container (read-only; no transcript text left the container). `chars/token = 3.4`, the codebase's own estimate. | session | lines | → | chars | → | saved | tokens | |---|---|---|---|---|---|---| | 0254b165 | 2522 | 1834 | 143,340 | 126,049 | 12.1% | ~5,085 | | 56cc4dee | 1904 | 1305 | 114,283 | 99,079 | 13.3% | ~4,471 | | a98d9eb8 | 1372 | 1042 | 93,409 | 85,012 | 9.0% | ~2,469 | | b7c7ae47 | 1407 | 1030 | 90,362 | 81,415 | 9.9% | ~2,631 | | d633ad3c | 1282 | 902 | 81,302 | 71,724 | 11.8% | ~2,817 | | 09168e79 | 1212 | 903 | 79,726 | 72,271 | 9.4% | ~2,192 | | 79b7b2d0 | 1107 | 838 | 70,197 | 63,599 | 9.4% | ~1,940 | | **total** | | | **672,619** | **599,149** | **10.9%** | **~21,608** | **The estimate was optimistic.** The issue projected ~35% of the prompt as per-line overhead and 10-15k tokens recovered. Actual: **10.9% of characters, ~1,900-5,100 tokens per session.** Line counts fell 24-30%, not "a large fraction" — speech clusters, but less than assumed. Still worth doing: it is free, lossless, and it is the margin that keeps a borderline session out of the chunked path. ## Deviations from the proposed fix **1. Runs are capped at 30 seconds.** Merging without a time bound collapses a five-minute GM monologue to one line at one timestamp — every beat drawn from it then shares a start time, so ordering *within* the monologue is lost and cited evidence can sit minutes from what it describes. Three existing beat tests failed on exactly that shape, which is how it surfaced. The cap costs ~1 percentage point (11.9% → 10.9%). **2. Per-block timestamps are NOT implemented, deliberately.** Under block stamps a citation can only say "somewhere in this 30 seconds", and a block holds several speakers — so "this actor spoke this line" degrades to "this actor spoke *near* this line". That is precisely the verification #332/#333/#334 exist to provide, and it is worth more than the remaining tokens. The issue's own hedge — "keeping exact per-line stamps only where the beat pipeline needs citations" — resolves to *everywhere*, because any line can be evidence. If the token pressure ever justifies revisiting this, the safe version is block headers in the prompt **plus** a per-line stamp retained on every line, which saves nothing — so realistically this half of the issue is closed as not-worth-doing rather than deferred. ## Consumers The format did not change, so `_TRANSCRIPT_LINE_RE`, the highlights prompt and the beat prompts needed no edit. Compaction applies to the **prompt only** — `session.transcript` is untouched, and every consumer downstream of `summarise()` reads the same compacted text, since the validator resolves stamps against it and indexing a different rendering than the model saw would fail every citation. `beat_service._LINE_RE` is now public `LINE_RE` — it is the shared definition of the line format, and re-declaring it per consumer is the coupling #335 set out to remove. Acceptance criteria: merged ✓, block timestamps ✗ (closed as harmful, reasoning above), measured ✓, consumers unchanged ✓, beat references still resolve ✓ (`test_every_remaining_stamp_still_indexes_to_a_real_line`).

rbrooks referenced this issue from a commit

2026-08-26 17:20:52 +00:00

feat(backend): show overlapping speech as a range, but only where it overlaps (#345)

claude-bot referenced this issue

2026-08-26 17:21:11 +00:00

[Backend] Emit start-end ranges and a deterministic tie-break in the merge #345

rbrooks referenced this issue from a commit

2026-08-27 15:40:12 +00:00

feat(backend): size extraction windows for attention, not for context (#423)

claude-bot commented

2026-08-27 23:27:14 +00:00

Author

Contributor

Verified against the acceptance criteria as amended by the 2026-08-26 comment, which is the right yardstick here.

Consecutive same-speaker runs merged — compact_transcript (audio_service.py:1591-1690), capped at 30 seconds per run so a monologue's internal ordering is not collapsed. Tested (tests/test_transcript_compaction.py:23-115).
Block timestamps — deliberately not implemented, and the comment records why: under block stamps a citation degrades from "this actor spoke this line" to "this actor spoke near this line", which is the exact verification #332–#334 exist to provide. Tokens are worth less than that. Recorded as a scope cut, not silently dropped.
Savings measured on real transcripts — 7 production transcripts, 10.9% of characters, ~1,900–5,100 tokens per session. Lower than the issue's own 35% estimate, and the comment says so rather than quietly restating the target.
Consumers in lockstep, beat evidence still resolves — test_every_remaining_stamp_still_indexes_to_a_real_line.

That measurement later paid for itself: it is what made the #345 conditional-range decision arguable from data rather than opinion.

Closing. Part of a full acceptance-criteria pass across the v4.0.0 milestone.

Verified against the acceptance criteria **as amended by the 2026-08-26 comment**, which is the right yardstick here. - **Consecutive same-speaker runs merged** — `compact_transcript` (`audio_service.py:1591-1690`), capped at 30 seconds per run so a monologue's internal ordering is not collapsed. Tested (`tests/test_transcript_compaction.py:23-115`). - **Block timestamps** — deliberately **not** implemented, and the comment records why: under block stamps a citation degrades from "this actor spoke this line" to "this actor spoke near this line", which is the exact verification #332–#334 exist to provide. Tokens are worth less than that. Recorded as a scope cut, not silently dropped. - **Savings measured on real transcripts** — 7 production transcripts, 10.9% of characters, ~1,900–5,100 tokens per session. Lower than the issue's own 35% estimate, and the comment says so rather than quietly restating the target. - **Consumers in lockstep, beat evidence still resolves** — `test_every_remaining_stamp_still_indexes_to_a_real_line`. That measurement later paid for itself: it is what made the #345 conditional-range decision arguable from data rather than opinion. Closing. Part of a full acceptance-criteria pass across the v4.0.0 milestone.

claude-bot closed this issue

2026-08-27 23:27:23 +00:00

rbrooks referenced this issue from a commit

2026-09-05 04:31:28 +00:00

feat(audio): anchor highlights to words

claude-bot referenced this issue

2026-09-09 06:43:53 +00:00

A summarisation-stage prompt of ~31k tokens was sent against a 22.9k budget; find the call and chunk it #575

claude-bot referenced this issue

2026-09-10 06:33:14 +00:00

Beat extraction reads a GM's answer to a table question as present-tense narration ("The party is currently located in Ironroot Hold") #584

rbrooks referenced this issue from a commit