[Backend][Data loss] Shortening transcript retention destroys transcripts with no trash, no grace and no warning

claude-bot commented

2026-09-17 17:50:06 +00:00

Collaborator

Found while scoping lane 2b of the surface mockups (#659) — specifically CAMP-13's retention override rows (CA-073, CA-074). Reported, not changed.

What happens

A GM changes Transcript retention from "retain indefinitely" to "retain N months" in campaign settings. On the next nightly enforce_retention run, every transcript in that campaign older than N months is destroyed:

session.transcript is set to None
every TranscriptSegment row for the session is deleted
the session's LoreExtractCache rows are purged

webapp/backend/app/tasks/reminder_tasks.py:4716-4757 (pass 3). There is no staging step, no trash, and no recovery: the structured segment rows hold the same words as the column (#335), so both copies go in the same statement.

The GM is told nothing. Not before, not during, not after. The form's only helper text is "Leave as default to use the instance-wide policy." (webapp/frontend/src/pages/CampaignDetail.jsx:2647-2667).

Why this is worse than it looks: audio is careful and transcripts are not

The same task treats audio with real caution, three layers deep:

	Audio	Transcript
Deleted inline when the window shortens?	No — moved to a trash (`audio_trashed_at = now`)	Yes, on the next tick
Grace period before the bytes go	`audio_trash_retention_days`, floored at 7 (`settings_service.py:228, 900-915`)	None
Warning before permanent deletion	Discord + in-app countdown, 3 days ahead (`AUDIO_DELETION_WARNING_LEAD_DAYS`, `reminder_tasks.py:4241, 4430-4546`)	None
Confirmation in the UI when shortening	None	None

Audio's docstring (reminder_tasks.py:4698-4704) is explicit that "every mode reaches deletion via the trash and its grace". That care simply was not extended to the transcript pass, and nothing in the code says the difference was intended.

The asymmetry is also backwards from what the data is worth. Audio can be re-transcribed while it exists. A transcript is the only durable record of what was said at the table once the audio has aged out — and under trash_after_processing, the default posture, the audio is already gone.

Trashed campaigns are protected; this case is not

Pass 3 already skips sessions whose campaign is trashed (reminder_tasks.py:4733-4735), with the comment "a restored campaign must come back whole, and a NULLed transcript is not recoverable by restoring the campaign." The hazard was understood. It just was not applied to the ordinary case of a GM shortening a window.

Where it lands in the redesign

CAMP-13 (Settings: Recording, retention & AI), being drawn now in lane 2b. D12 is directly on point: "a confirmation that states the consequence for other irreversible actions." The mockup will draw that confirmation, naming how many transcripts the change would destroy. But the drawing is the front half of the fix — the backend has no trash to fall back on if someone confirms by accident, or if the count is computed wrong.

Suggested shape

A confirmation in the UI, stating the count. The GM should see "This will permanently delete 14 transcripts from before June 2026" before saving, not after. This is the lane 2b mockup's job and is happening anyway.
Give transcripts the same trash and grace as audio. A transcript_trashed_at alongside audio_trashed_at, purged by the existing grace sweep. This is the substantive fix: it turns an irreversible accident into a recoverable one, and it reuses machinery that already exists.
Warn before permanent deletion, reusing warn_before_audio_deletion's shape. Cheap once (2) exists, pointless before it.
A test that shortening the window trashes rather than destroys, in the shape of the existing retention tests.

(1) alone is worth doing and is not sufficient — a confirmation prevents the careless case, not the mistaken one.

Not in scope here

Whether a campaign should be able to set a longer retention than the instance is a separate question, answered by the owner on 2026-09-17 while scoping this lane: the instance setting becomes a default, the admin gains an absolute cap, and a campaign may land anywhere between the two. That is its own issue against ADMIN-5/ADMIN-6 and CAMP-13.

Found by reading the code; not run against any instance, and no data was touched.

Found while scoping lane 2b of the surface mockups (#659) — specifically `CAMP-13`'s retention override rows (`CA-073`, `CA-074`). **Reported, not changed.** ## What happens A GM changes **Transcript retention** from "retain indefinitely" to "retain N months" in campaign settings. On the next nightly `enforce_retention` run, every transcript in that campaign older than N months is destroyed: - `session.transcript` is set to `None` - every `TranscriptSegment` row for the session is deleted - the session's `LoreExtractCache` rows are purged `webapp/backend/app/tasks/reminder_tasks.py:4716-4757` (pass 3). There is no staging step, no trash, and no recovery: the structured segment rows hold the same words as the column (#335), so both copies go in the same statement. The GM is told nothing. Not before, not during, not after. The form's only helper text is *"Leave as default to use the instance-wide policy."* (`webapp/frontend/src/pages/CampaignDetail.jsx:2647-2667`). ## Why this is worse than it looks: audio is careful and transcripts are not The same task treats audio with real caution, three layers deep: | | Audio | Transcript | |---|---|---| | Deleted inline when the window shortens? | No — moved to a trash (`audio_trashed_at = now`) | **Yes, on the next tick** | | Grace period before the bytes go | `audio_trash_retention_days`, floored at 7 (`settings_service.py:228, 900-915`) | **None** | | Warning before permanent deletion | Discord + in-app countdown, 3 days ahead (`AUDIO_DELETION_WARNING_LEAD_DAYS`, `reminder_tasks.py:4241, 4430-4546`) | **None** | | Confirmation in the UI when shortening | None | **None** | Audio's docstring (`reminder_tasks.py:4698-4704`) is explicit that *"every mode reaches deletion via the trash and its grace"*. That care simply was not extended to the transcript pass, and nothing in the code says the difference was intended. The asymmetry is also backwards from what the data is worth. Audio can be re-transcribed while it exists. A transcript is the only durable record of what was said at the table once the audio has aged out — and under `trash_after_processing`, the default posture, the audio is *already gone*. ## Trashed campaigns are protected; this case is not Pass 3 already skips sessions whose campaign is trashed (`reminder_tasks.py:4733-4735`), with the comment *"a restored campaign must come back whole, and a NULLed transcript is not recoverable by restoring the campaign."* The hazard was understood. It just was not applied to the ordinary case of a GM shortening a window. ## Where it lands in the redesign `CAMP-13` (Settings: Recording, retention & AI), being drawn now in lane 2b. **D12** is directly on point: *"a confirmation that states the consequence for other irreversible actions."* The mockup will draw that confirmation, naming how many transcripts the change would destroy. But the drawing is the front half of the fix — the backend has no trash to fall back on if someone confirms by accident, or if the count is computed wrong. ## Suggested shape 1. **A confirmation in the UI, stating the count.** The GM should see "This will permanently delete 14 transcripts from before June 2026" before saving, not after. This is the lane 2b mockup's job and is happening anyway. 2. **Give transcripts the same trash and grace as audio.** A `transcript_trashed_at` alongside `audio_trashed_at`, purged by the existing grace sweep. This is the substantive fix: it turns an irreversible accident into a recoverable one, and it reuses machinery that already exists. 3. **Warn before permanent deletion**, reusing `warn_before_audio_deletion`'s shape. Cheap once (2) exists, pointless before it. 4. **A test that shortening the window trashes rather than destroys**, in the shape of the existing retention tests. (1) alone is worth doing and is not sufficient — a confirmation prevents the careless case, not the mistaken one. ## Not in scope here Whether a campaign should be able to set a *longer* retention than the instance is a separate question, answered by the owner on 2026-09-17 while scoping this lane: the instance setting becomes a default, the admin gains an absolute cap, and a campaign may land anywhere between the two. That is its own issue against `ADMIN-5`/`ADMIN-6` and `CAMP-13`. Found by reading the code; not run against any instance, and no data was touched.

rbrooks referenced this issue from a commit

2026-09-17 18:20:23 +00:00

feat(feedback-site): lane 2b — the rest of Settings (#659)

claude-bot referenced this issue

2026-09-17 18:20:46 +00:00

Lane 2b — the rest of Settings (#659) #682

rbrooks referenced this issue from a commit

2026-09-17 18:55:43 +00:00

Merge PR #682: lane 2b — the rest of Settings (#659)

claude-bot referenced this issue

2026-09-17 18:58:52 +00:00

[UX] Mock up every catalogued surface, for players and GMs, before building #659

rbrooks referenced this issue from a commit

2026-09-23 10:18:16 +00:00

feat(webapp): Settings — Discord, and Recording, retention & AI (#729, lane 3)

claude-bot commented

2026-10-01 19:02:23 +00:00

Author

Collaborator

Status check against main, 1 Oct: half built, staying open.

Built (the warning). Before a GM saves a shorter transcript window, Settings → Recording, retention & AI asks the server what the nightly sweep would take (GET /campaigns/{id}/retention-preview, which uses the sweep's own selection and writes nothing). The dialog names how many sessions and which, says a deleted transcript is gone with no bin behind it, and offers to export first. Tests: test_retention_preview.py.

Not built (the safety net).

The nightly sweep still destroys an expired transcript outright: enforce_retention's third pass sets session.transcript = None and deletes the segments. There is no transcript_trashed_at anywhere.
No warning before that deletion, as audio has.
No test that shortening the window bins rather than destroys.
The confirmation is a step in the page; the PATCH itself saves at once. Whether a direct API call should be refused without it is worth deciding here too.

What remains, concretely: transcript_trashed_at with a migration; the destruction moved out of pass 3 into a grace sweep shaped like audio's; a campaign restore bringing transcripts back; a pre-deletion warning like warn_before_audio_deletion; the test; and the dialog's "There is no bin behind it" changed once that is no longer true. This is a contained piece of work that doesn't need a product decision, so it is in the queue.

Status check against main, 1 Oct: half built, staying open. **Built (the warning).** Before a GM saves a shorter transcript window, Settings → Recording, retention & AI asks the server what the nightly sweep would take (`GET /campaigns/{id}/retention-preview`, which uses the sweep's own selection and writes nothing). The dialog names how many sessions and which, says a deleted transcript is gone with no bin behind it, and offers to export first. Tests: `test_retention_preview.py`. **Not built (the safety net).** - The nightly sweep still destroys an expired transcript outright: `enforce_retention`'s third pass sets `session.transcript = None` and deletes the segments. There is no `transcript_trashed_at` anywhere. - No warning before that deletion, as audio has. - No test that shortening the window bins rather than destroys. - The confirmation is a step in the page; the PATCH itself saves at once. Whether a direct API call should be refused without it is worth deciding here too. **What remains, concretely:** `transcript_trashed_at` with a migration; the destruction moved out of pass 3 into a grace sweep shaped like audio's; a campaign restore bringing transcripts back; a pre-deletion warning like `warn_before_audio_deletion`; the test; and the dialog's "There is no bin behind it" changed once that is no longer true. This is a contained piece of work that doesn't need a product decision, so it is in the queue.

rbrooks referenced this issue from a commit

2026-10-01 19:55:23 +00:00

WIP: the bin's readers, restore, warning and tests (#680)

rbrooks referenced this issue from a commit

2026-10-01 19:55:23 +00:00

WIP: the dialog says the bin, and a GM can restore a transcript (#680)

rbrooks referenced this issue from a commit

2026-10-01 19:55:23 +00:00

WIP: the export's bin rule covers a binned transcript (#680)

rbrooks referenced this issue from a commit

2026-10-01 19:55:23 +00:00

WIP: docs, changelog, catalogue row for the transcript bin (#680)

rbrooks referenced this issue from a commit

2026-10-01 19:55:23 +00:00

WIP: the admin's cap question says the bin (#680)

rbrooks referenced this issue from a commit

2026-10-01 19:55:23 +00:00

An expired transcript goes to the bin first, with a grace period and a warning (#680)

claude-bot referenced this issue

2026-10-01 20:16:35 +00:00

Dev deploy, 1 Oct: lane 11, the R3 fixes and today's issue fixes #1004

claude-bot referenced this issue from a pull request that will close it,

2026-10-01 20:22:38 +00:00

An expired transcript goes to the bin first, with a grace period, a warning and Restore (#680) #1005

claude-bot referenced this issue

2026-10-01 20:38:05 +00:00

Shortening the transcript window is an ordinary confirmation now it can be undone (Q140 / R2) #1010

claude-bot closed this issue

2026-10-01 20:47:37 +00:00

claude-bot referenced this issue from a commit

2026-10-01 20:47:37 +00:00

WIP: an expired transcript goes to the bin first (#680)

claude-bot referenced this issue from a commit

2026-10-01 20:47:38 +00:00

Merge PR #1005: an expired transcript goes to the bin first, with a grace period, a warning and Restore (#680)

claude-bot commented

2026-10-01 20:48:33 +00:00

Author

Collaborator

Done: PR #1005 merged (main 20942dc9, CI green on 09dde28f).

An expired transcript now goes to a bin first (sessions.transcript_trashed_at), waits as long as audio does, the GMs are warned three days before, and a GM restores it from the session's Review tab. While binned it is withheld from the session payload, both searches, the transcript tools, every model prompt and the export; tests/test_transcript_bin_readers.py fails on a new unclassified read. Migration b680d1e2f3a4.

The owner's answers (2026-10-01):

Q139 (R1): any GM of the campaign can restore, as built. D267.
Q140 (R2): the confirmation for shortening the window becomes an ordinary one, keeping the offer to export first. That is a follow-up: #1010. D268.

Left for later: the prototype pages still draw the old "no bin" wording (lane1c, lane2b, lane7, lane8); webapp/CLAUDE.md says the sweep has three passes and it now has four (told to the owner, who edits that file). Not on dev until the next deploy.

**Done: PR #1005 merged** (main `20942dc9`, CI green on `09dde28f`). An expired transcript now goes to a bin first (`sessions.transcript_trashed_at`), waits as long as audio does, the GMs are warned three days before, and a GM restores it from the session's Review tab. While binned it is withheld from the session payload, both searches, the transcript tools, every model prompt and the export; `tests/test_transcript_bin_readers.py` fails on a new unclassified read. Migration `b680d1e2f3a4`. **The owner's answers (2026-10-01):** - Q139 (R1): any GM of the campaign can restore, as built. D267. - Q140 (R2): the confirmation for shortening the window becomes an ordinary one, keeping the offer to export first. That is a follow-up: **#1010**. D268. **Left for later:** the prototype pages still draw the old "no bin" wording (`lane1c`, `lane2b`, `lane7`, `lane8`); `webapp/CLAUDE.md` says the sweep has three passes and it now has four (told to the owner, who edits that file). Not on dev until the next deploy.

rbrooks referenced this issue from a commit

2026-10-01 20:53:22 +00:00

Shortening the transcript window is an ordinary confirmation (#1010)

rbrooks referenced this issue from a commit

2026-10-01 21:36:14 +00:00

Recompose reads no binned transcript for its pronoun lookup (#999, #680)

rbrooks referenced this issue from a commit

2026-10-01 21:37:14 +00:00

The bell's Show older row is IS-466: #680 took IS-465 (#1000)

claude-bot referenced this issue

2026-10-01 21:57:23 +00:00

The writing models are told people's pronouns, players' and characters' apart (#999) #1013