Retention counts from a session's first processing, not its last edit (#791) #800

Merged

claude-bot merged 2 commits from build/791-retention-anchor into main

2026-09-24 03:37:22 +00:00

claude-bot commented

2026-09-24 03:08:45 +00:00

Collaborator

Closes #791 (R1: the owner chose "from when the audio was first processed").

The bug

Both retention windows counted from transcript_updated_at, which records when the transcript was last touched. A member erase rewrites it on every session it scrubs, and so does a reprocess. So erasing one member from a 25-day-old session under a 30-day policy kept everyone else's audio and words for another 30 days. A privacy control caused more to be kept.

The fix

New write-once column sessions.audio_processed_at. process_audio's success path stamps it only while it's NULL, so neither an erase nor a reprocess moves it.
Migration e1f2a3b4c5d6 (on top of d0e1f2a3b4c5, single head):
- Adds the column with raw DDL.
- Backfills it from transcript_updated_at, which is exactly the anchor each row had at upgrade time. Nothing is kept or deleted differently on upgrade day.
- Sessions already erased or reprocessed before the upgrade keep the later date they were given. Their true first-processing time was never recorded, and guessing it would delete things the GM had been told were kept.
- Downgrade drops the column.
app/retention.py:
- first_processed_at() reads the stamp. It falls back to transcript_updated_at only for an unstamped import from a pre-#791 export.
- in_flight(): a session in processing is never caught, so the sweep can't bin audio out from under its own reprocess. A run that dies is moved to failed after 8 hours by unstick_lost_tasks, so this can't hold audio for ever.
- The nightly sweep and the preview still share select_retention_catch.
Export and import carry the new column.

Tests

test_retention_anchor.py:
- The per-row rules.
- The process_audio seam: the first success stamps, a reprocess doesn't move the stamp, and a failure stamps nothing.
- A real run_member_erasure end to end: on day 25 of 30 nothing is caught; seven days later the preview and the sweep both catch it.
The preview/sweep agreement test gains erased, reprocessed, failed-reprocess and in-flight seeds.
test_audio_processed_at_migration.py runs the real upgrade()/downgrade(): backfill, nothing else moves, the round trip, and idempotence.
Checked that the tests fail against main's retention.py.
Full backend suite on the merged tree: 4135 passed, 13 skipped. Ruff 0.4.4 format and check are clean. The migration was also run for real on postgres:16 with seeded rows.

Follow-ups filed

#798: a failed run's clock still restarts on any write. It needs its own write-once "audio received" stamp.
#797: the erase dialog's "regenerate summaries" never runs. Owner decision needed.

🤖 Generated with Claude Code

Closes #791 (R1: the owner chose "from when the audio was first processed"). ## The bug Both retention windows counted from `transcript_updated_at`, which records when the transcript was last *touched*. A member erase rewrites it on every session it scrubs, and so does a reprocess. So erasing one member from a 25-day-old session under a 30-day policy kept everyone else's audio and words for another 30 days. A privacy control caused more to be kept. ## The fix - **New write-once column `sessions.audio_processed_at`.** `process_audio`'s success path stamps it only while it's NULL, so neither an erase nor a reprocess moves it. - **Migration `e1f2a3b4c5d6`** (on top of `d0e1f2a3b4c5`, single head): - Adds the column with raw DDL. - Backfills it from `transcript_updated_at`, which is exactly the anchor each row had at upgrade time. **Nothing is kept or deleted differently on upgrade day.** - Sessions already erased or reprocessed before the upgrade keep the later date they were given. Their true first-processing time was never recorded, and guessing it would delete things the GM had been told were kept. - Downgrade drops the column. - **`app/retention.py`:** - `first_processed_at()` reads the stamp. It falls back to `transcript_updated_at` only for an unstamped import from a pre-#791 export. - `in_flight()`: a session in `processing` is never caught, so the sweep can't bin audio out from under its own reprocess. A run that dies is moved to `failed` after 8 hours by `unstick_lost_tasks`, so this can't hold audio for ever. - The nightly sweep and the preview still share `select_retention_catch`. - Export and import carry the new column. ## Tests - `test_retention_anchor.py`: - The per-row rules. - The `process_audio` seam: the first success stamps, a reprocess doesn't move the stamp, and a failure stamps nothing. - A real `run_member_erasure` end to end: on day 25 of 30 nothing is caught; seven days later the preview and the sweep both catch it. - The preview/sweep agreement test gains erased, reprocessed, failed-reprocess and in-flight seeds. - `test_audio_processed_at_migration.py` runs the real `upgrade()`/`downgrade()`: backfill, nothing else moves, the round trip, and idempotence. - Checked that the tests fail against main's `retention.py`. - Full backend suite on the merged tree: 4135 passed, 13 skipped. Ruff 0.4.4 format and check are clean. The migration was also run for real on postgres:16 with seeded rows. ## Follow-ups filed - #798: a failed run's clock still restarts on any write. It needs its own write-once "audio received" stamp. - #797: the erase dialog's "regenerate summaries" never runs. Owner decision needed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 2 commits

2026-09-24 03:08:45 +00:00

fix(backend): retention counts from a session's first processing, not its last edit (#791 , R1) 1089466663

Both retention windows anchored on sessions.transcript_updated_at, which a
member erase rewrites on every session it scrubs and a reprocess rewrites
too. Erasing one member therefore restarted the audio and transcript clocks
for everybody else in those sessions — a privacy control that kept more —
and a session mid-reprocess, still carrying its old anchor, could have its
audio binned out from under the run.

- New write-once column sessions.audio_processed_at, stamped by
  process_audio's success path only while it is NULL. Neither a reprocess
  nor an erase moves it. The erase and reprocess still write
  transcript_updated_at, which the upload guard, the process_audio guard,
  the session page and the attention feed read.
- app/retention.py: both windows measure from first_processed_at()
  (audio_processed_at, falling back to transcript_updated_at only for an
  unstamped row, i.e. one imported from a pre-#791 export). A failed run
  that never succeeded still counts from updated_at, unchanged. Option A's
  transcribed_successfully() keys on the same helper; its meaning is
  unchanged. A session in `processing` is never caught on either side.
- Migration e1f2a3b4c5d6 backfills audio_processed_at from
  transcript_updated_at, i.e. each row's anchor as of the upgrade, so
  nothing is caught earlier or later on upgrade day (bar a session
  mid-processing, which now waits for its run).
- Export/import carry the stamp.
- Tests: the erase end to end through the preview and the sweep; process_audio
  stamps once and a reprocess does not move it; processing never caught; the
  agreement test's seeds gain erased, reprocessed, failed-reprocess and
  mid-reprocess sessions; the migration's backfill, downgrade and re-run.
  The one-head pin moves to the new migration's test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into build/791-retention-anchor

CI / Bot/backend version sync (pull_request) Successful in 1m9s

Details

CI / Backend lint (ruff) (pull_request) Successful in 1m34s

Details

CI / Docker image build (pull_request) Successful in 35s

Details

CI / Does this change touch the recording path? (pull_request) Successful in 18s

Details

CI / Synthetic session harness (no GPU, no LLM) (pull_request) Has been skipped

Details

CI / Feedback site tests (pull_request) Successful in 2m7s