rbrooks/Quest-Board

Watch

Fork 0

[Backend] Warn and show a countdown before retention deletes a failed transcription's only audio copy #402

New issue

Closed

opened 2026-08-25 20:44:41 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-08-25 20:44:41 +00:00

Collaborator

Severity: HIGH

Found in the August 2026 session lifecycle review (#319).

A session whose transcription failed has its only surviving audio — the raw recording itself, since no transcript was ever produced — permanently deleted by the default retention policy after 7 days, with no notification sent to the GM and no way for the GM to even see the countdown before it happens on the page where they'd notice it.

Evidence

webapp/backend/app/services/settings_service.py:129 (_DEFAULT_AUDIO_RETENTION_DAYS = 7) combined with the default mode AUDIO_RETENTION_DELETE_AFTER_PROCESSING (:89) — this is the out-of-the-box policy for every new instance/campaign that doesn't override it.
webapp/backend/app/tasks/reminder_tasks.py:2678-2712 (enforce_retention, pass 2) — for delete_after_processing mode, this pass only ever reaches a session that failed processing (a successful run's audio is already deleted inline by apply_post_processing_audio_retention at the moment it succeeds). The anchor for a failed run is session.updated_at (:2705-2707), so it's swept on the same 7-day window as everything else — a debug grace period, not a deliberate retention decision for a failed job's only copy.
webapp/backend/app/routers/admin.py:88-93 — the permanent_deletion_at computation that would let someone see a countdown exists only on the admin recordings endpoint; there is no equivalent surfaced on the GM-facing session page.
No notification task fires before this deletion — there is no Celery task or notification call referencing permanent_deletion_at or a "your audio will be deleted" message anywhere in reminder_tasks.py.

Failure scenario
A hosted tenant's Whisper endpoint is briefly misconfigured. Friday's session fails transcription (audio_processing_status = 'failed'), and the GM — heading into a week off — doesn't notice. Day 8 rolls around: enforce_retention sweeps the failed session's audio directory because it's the only artifact still on disk and the 7-day clock has expired. The recording is gone. Since transcription never succeeded, no transcript exists either — the entire session is unrecoverable, with no warning ever sent.

Proposed fix
Never delete a failed-processing session's audio without first delivering a notification to the GM (Discord DM or in-app) with a clear countdown, and surface permanent_deletion_at on the session detail page itself (not just the admin console) once a session is failed and its clock has started. Consider also lengthening the default window specifically for the failed case (distinct from the successful-and-approved case, where deletion is intentional and low-risk) since a failed run's audio is the only copy of anything for that session.

Acceptance criteria

A GM-visible notification is sent with adequate lead time before a failed session's audio is permanently deleted by retention.
The session detail page (not just the admin recordings list) shows the computed permanent_deletion_at for any session with audio pending deletion.
The failed-processing retention window is independently configurable from the general delete_after_processing window (or otherwise deliberately widened), since it guards the only remaining copy of the recording.
Regression test: a failed session approaching its retention cutoff triggers the notification before deletion actually runs.

**Severity: HIGH** Found in the August 2026 session lifecycle review (#319). A session whose transcription failed has its only surviving audio — the raw recording itself, since no transcript was ever produced — permanently deleted by the default retention policy after 7 days, with no notification sent to the GM and no way for the GM to even see the countdown before it happens on the page where they'd notice it. **Evidence** - `webapp/backend/app/services/settings_service.py:129` (`_DEFAULT_AUDIO_RETENTION_DAYS = 7`) combined with the default mode `AUDIO_RETENTION_DELETE_AFTER_PROCESSING` (`:89`) — this is the out-of-the-box policy for every new instance/campaign that doesn't override it. - `webapp/backend/app/tasks/reminder_tasks.py:2678-2712` (`enforce_retention`, pass 2) — for `delete_after_processing` mode, this pass only ever reaches a session that failed processing (a successful run's audio is already deleted inline by `apply_post_processing_audio_retention` at the moment it succeeds). The anchor for a failed run is `session.updated_at` (`:2705-2707`), so it's swept on the same 7-day window as everything else — a debug grace period, not a deliberate retention decision for a failed job's only copy. - `webapp/backend/app/routers/admin.py:88-93` — the `permanent_deletion_at` computation that would let someone see a countdown exists only on the admin recordings endpoint; there is no equivalent surfaced on the GM-facing session page. - No notification task fires before this deletion — there is no Celery task or notification call referencing `permanent_deletion_at` or a "your audio will be deleted" message anywhere in `reminder_tasks.py`. **Failure scenario** A hosted tenant's Whisper endpoint is briefly misconfigured. Friday's session fails transcription (`audio_processing_status = 'failed'`), and the GM — heading into a week off — doesn't notice. Day 8 rolls around: `enforce_retention` sweeps the failed session's audio directory because it's the only artifact still on disk and the 7-day clock has expired. The recording is gone. Since transcription never succeeded, no transcript exists either — the entire session is unrecoverable, with no warning ever sent. **Proposed fix** Never delete a failed-processing session's audio without first delivering a notification to the GM (Discord DM or in-app) with a clear countdown, and surface `permanent_deletion_at` on the session detail page itself (not just the admin console) once a session is failed and its clock has started. Consider also lengthening the default window specifically for the failed case (distinct from the successful-and-approved case, where deletion is intentional and low-risk) since a failed run's audio is the only copy of anything for that session. **Acceptance criteria** - [ ] A GM-visible notification is sent with adequate lead time before a failed session's audio is permanently deleted by retention. - [ ] The session detail page (not just the admin recordings list) shows the computed `permanent_deletion_at` for any session with audio pending deletion. - [ ] The failed-processing retention window is independently configurable from the general `delete_after_processing` window (or otherwise deliberately widened), since it guards the only remaining copy of the recording. - [ ] Regression test: a failed session approaching its retention cutoff triggers the notification before deletion actually runs.

claude-bot added this to the v4.1.0 - Data Durability & Recovery milestone

2026-08-25 20:44:41 +00:00

claude-bot added the

bug

backend

labels

2026-08-25 20:44:41 +00:00

claude-bot commented

2026-08-29 01:21:14 +00:00

Author

Collaborator

Partially addressed by #427 (d4881b9 on fix/v4.0.1-retention-safety) — noting the change in shape, not claiming this is done.

The "countdown" half now exists for free. Previously a failed transcription's audio was deleted directly by the retention sweep when its window expired: no trash state, no audio_trashed_at, so nothing for the Admin recordings page to compute permanent_deletion_at from. There was no countdown to show because there was no interval to count down.

#427 made every retention mode reach deletion through the trash. So an expiring window on a failed session now:

sets audio_processing_status = trashed and stamps audio_trashed_at,
which makes it appear in the Admin recordings list with a real permanent_deletion_at,
and buys it a further 7-day grace (floored in code) before anything is removed.

So the GM gets both a visible countdown and considerably more time — a failed session's only audio copy now survives retention_window + 7 days instead of vanishing on a Beat tick.

Still outstanding: the "warn" half. Nothing proactively tells the GM. They have to go and look at the Admin page. If the intent is a notification — a Discord message, an email, a dashboard banner when a failed session is approaching permanent deletion — that is untouched and is the substantive remainder of this issue.

Worth rescoping the title/body to just the warning, since the countdown part is no longer missing.

**Partially addressed by #427** (`d4881b9` on `fix/v4.0.1-retention-safety`) — noting the change in shape, not claiming this is done. The "countdown" half now exists for free. Previously a `failed` transcription's audio was deleted **directly** by the retention sweep when its window expired: no trash state, no `audio_trashed_at`, so nothing for the Admin recordings page to compute `permanent_deletion_at` from. There was no countdown to show because there was no interval to count down. #427 made every retention mode reach deletion through the trash. So an expiring window on a failed session now: 1. sets `audio_processing_status = trashed` and stamps `audio_trashed_at`, 2. which makes it appear in the Admin recordings list with a real `permanent_deletion_at`, 3. and buys it a further 7-day grace (floored in code) before anything is removed. So the GM gets both a visible countdown and considerably more time — a failed session's only audio copy now survives `retention_window + 7 days` instead of vanishing on a Beat tick. **Still outstanding: the "warn" half.** Nothing proactively tells the GM. They have to go and look at the Admin page. If the intent is a notification — a Discord message, an email, a dashboard banner when a failed session is approaching permanent deletion — that is untouched and is the substantive remainder of this issue. Worth rescoping the title/body to just the warning, since the countdown part is no longer missing.

claude-bot referenced this issue

2026-08-29 05:58:03 +00:00

v4.0.1 — retention safety and secrets at rest #435

rbrooks referenced this issue from a commit

2026-08-31 03:23:48 +00:00

feat(retention): warn before a recording is deleted for good (#402)

claude-bot referenced this issue from a pull request that will close it,

2026-08-31 03:24:07 +00:00

feat(retention): warn before a recording is deleted for good (#402) #458

claude-bot closed this issue

2026-08-31 03:43:02 +00:00

claude-bot referenced this issue from a commit

2026-08-31 03:43:03 +00:00

Merge PR #458: warn before a recording is deleted for good (#402)

claude-bot commented

2026-08-31 03:43:20 +00:00

Author

Collaborator

Merged in PR #458 (CI green). Closing.

Acceptance criteria:

A GM-visible notification with adequate lead time — a daily task posts to the campaign's Discord channel 3 days out, once per trip through the trash.
The session detail page shows permanent_deletion_at, not just the admin recordings list.
The failed-processing window is deliberately widened — by #427, not by a new setting. See below.
Regression test: a failed session approaching its cutoff triggers the notification before deletion runs.

The one criterion resolved differently from the issue text

The issue asks for the failed case to be "independently configurable from the general delete_after_processing window (or otherwise deliberately widened)". #427 already did the widening: every mode reaches deletion through the trash, with a 7-day grace floored in code rather than defaulted, so a failed session's audio survives retention_window + 7 days. The issue's premise — that it "vanishes on a Beat tick" — was true when it was filed and is no longer.

A separate knob was rejected rather than forgotten: more retention settings means more ways to misconfigure the one artefact in this system that cannot be regenerated. Agreed with you before building.

Worth knowing for future retention work

permanent_deletion_at was computed inline in the admin router while the sweeper derived the same boundary from the opposite direction. Rather than add a third copy in the warning task, it moved to app/retention.py and all three now read it — a countdown that disagrees with the sweep is worse than no countdown, because it gets believed.

Two details that are load-bearing rather than incidental:

The sent-marker is written before the publish. A failed publish is retried by the stream; a publish that succeeded while the marker write failed would re-announce tomorrow, and rarity is the entire mechanism.
The marker is cleared wherever audio leaves the trash. A plain boolean would mark a recording warned forever and go silent on the trip that actually ends in deletion.

Mutation-checked both guards independently. 1,470 backend, 457 frontend, 220 bot tests pass.

Merged in PR #458 (CI green). Closing. **Acceptance criteria:** - [x] A GM-visible notification with adequate lead time — a daily task posts to the campaign's Discord channel 3 days out, once per trip through the trash. - [x] The session detail page shows `permanent_deletion_at`, not just the admin recordings list. - [x] The failed-processing window is deliberately widened — **by #427**, not by a new setting. See below. - [x] Regression test: a failed session approaching its cutoff triggers the notification before deletion runs. ## The one criterion resolved differently from the issue text The issue asks for the failed case to be "independently configurable from the general `delete_after_processing` window (or otherwise deliberately widened)". #427 already did the widening: every mode reaches deletion through the trash, with a 7-day grace **floored in code** rather than defaulted, so a failed session's audio survives `retention_window + 7 days`. The issue's premise — that it "vanishes on a Beat tick" — was true when it was filed and is no longer. A separate knob was rejected rather than forgotten: more retention settings means more ways to misconfigure the one artefact in this system that cannot be regenerated. Agreed with you before building. ## Worth knowing for future retention work `permanent_deletion_at` was computed inline in the admin router while the sweeper derived the same boundary from the opposite direction. Rather than add a third copy in the warning task, it moved to `app/retention.py` and all three now read it — **a countdown that disagrees with the sweep is worse than no countdown, because it gets believed.** Two details that are load-bearing rather than incidental: - The sent-marker is written **before** the publish. A failed publish is retried by the stream; a publish that succeeded while the marker write failed would re-announce tomorrow, and rarity is the entire mechanism. - The marker is cleared wherever audio leaves the trash. A plain boolean would mark a recording warned forever and go silent on the trip that actually ends in deletion. Mutation-checked both guards independently. 1,470 backend, 457 frontend, 220 bot tests pass.

claude-bot referenced this issue

2026-08-31 22:09:49 +00:00

feat(ops): ship backups off-host and cover the audio volume (#414) #472

claude-bot referenced this issue

2026-08-31 22:43:03 +00:00

[Ops] Move backups off-host and cover the audio volume #414