[Ops] Document restore procedure, add off-host backup copies, and back up the media volume #80

New issue

Closed

opened 2026-07-14 19:45:04 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-07-14 19:45:04 +00:00

Contributor

Context

Scheduled database backups exist and work: run_scheduled_backup (webapp/backend/app/tasks/reminder_tasks.py:1832) and make backup-now write pg_dump custom-format dumps to the backups Docker volume, with retention. But the backup story stops there:

No restore procedure is documented anywhere — docs/OPERATIONS.md covers taking backups only.
The backups volume lives on the same host/disk as postgres_data, so a single disk failure loses the database and every backup of it.
The media volume (uploaded lore/character images — permanent user data; declared at docker-compose.yml:137, mounted rw into backend at docker-compose.yml:48 and ro into frontend at docker-compose.yml:103) is not backed up at all.

Current behavior

A dump nobody has ever restored is an untested backup; an operator facing a dead host today would have to improvise pg_restore flags, service ordering, and Alembic state under pressure — and would discover all uploaded media is simply gone.

Fix / Spec

Restore runbook in docs/OPERATIONS.md, end to end:
- Stop the stack (or at minimum backend/worker/beat/bot) so nothing writes during restore.
- pg_restore of the custom-format dump into a fresh database (exact command incl. --clean/--create decision, connecting as the app user vs postgres superuser).
- Start ordering afterwards (db → redis → backend → rest).
- Post-restore verification: alembic current matches alembic heads; app boots; a spot-check query.
Off-host copies: document a pull-based pattern (rsync or rclone cron job on another machine copying the contents of the backups volume), and/or add an optional rclone push step to run_scheduled_backup gated entirely by env config (no-op when unset). Either is acceptable; document whichever is implemented.
Media backup: include the media volume in the backup story — either extend the scheduled backup task to tar the media directory alongside the dump (it is mounted at /app/media in the backend container), or document a separate volume-backup step with the same retention/off-host treatment. Decide and implement one.
State explicitly in the docs that audio_temp is transient scratch space and deliberately excluded from backups.

Acceptance criteria

An operator can rehearse a full restore (db + media) from the docs alone, with no improvisation, on a fresh host.
Media files survive a simulated host loss when the documented setup is followed.
Docs state the off-host copy pattern and the audio_temp exclusion.

References

webapp/backend/app/tasks/reminder_tasks.py:1832 (run_scheduled_backup)
docker-compose.yml:48, docker-compose.yml:103, docker-compose.yml:137 (media volume)
docs/OPERATIONS.md (backup section)

Filed from the July 2026 full-project review.

## Context Scheduled database backups exist and work: `run_scheduled_backup` (`webapp/backend/app/tasks/reminder_tasks.py:1832`) and `make backup-now` write pg_dump custom-format dumps to the `backups` Docker volume, with retention. But the backup story stops there: - No restore procedure is documented anywhere — `docs/OPERATIONS.md` covers *taking* backups only. - The `backups` volume lives on the same host/disk as `postgres_data`, so a single disk failure loses the database **and** every backup of it. - The `media` volume (uploaded lore/character images — permanent user data; declared at `docker-compose.yml:137`, mounted rw into backend at `docker-compose.yml:48` and ro into frontend at `docker-compose.yml:103`) is not backed up at all. ## Current behavior A dump nobody has ever restored is an untested backup; an operator facing a dead host today would have to improvise `pg_restore` flags, service ordering, and Alembic state under pressure — and would discover all uploaded media is simply gone. ## Fix / Spec 1. **Restore runbook** in `docs/OPERATIONS.md`, end to end: - Stop the stack (or at minimum backend/worker/beat/bot) so nothing writes during restore. - `pg_restore` of the custom-format dump into a fresh database (exact command incl. `--clean`/`--create` decision, connecting as the app user vs postgres superuser). - Start ordering afterwards (db → redis → backend → rest). - Post-restore verification: `alembic current` matches `alembic heads`; app boots; a spot-check query. 2. **Off-host copies**: document a pull-based pattern (rsync or rclone cron job on another machine copying the contents of the `backups` volume), and/or add an optional rclone push step to `run_scheduled_backup` gated entirely by env config (no-op when unset). Either is acceptable; document whichever is implemented. 3. **Media backup**: include the `media` volume in the backup story — either extend the scheduled backup task to tar the media directory alongside the dump (it is mounted at `/app/media` in the backend container), or document a separate volume-backup step with the same retention/off-host treatment. Decide and implement one. 4. State explicitly in the docs that `audio_temp` is transient scratch space and deliberately excluded from backups. ## Acceptance criteria - [ ] An operator can rehearse a full restore (db + media) from the docs alone, with no improvisation, on a fresh host. - [ ] Media files survive a simulated host loss when the documented setup is followed. - [ ] Docs state the off-host copy pattern and the audio_temp exclusion. ## References - `webapp/backend/app/tasks/reminder_tasks.py:1832` (`run_scheduled_backup`) - `docker-compose.yml:48`, `docker-compose.yml:103`, `docker-compose.yml:137` (media volume) - `docs/OPERATIONS.md` (backup section) _Filed from the July 2026 full-project review._

claude-bot added this to the v3.4.0 - Reliability, Operations & Self-Hosting milestone

2026-07-14 19:45:13 +00:00

claude-bot added the

backend

enhancement

labels

2026-07-14 19:45:46 +00:00