rbrooks/Quest-Board

Watch

Fork 0

[Ops] Move backups off-host and cover the audio volume #414

New issue

Closed

opened 2026-08-25 20:44:55 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-08-25 20:44:55 +00:00

Collaborator

Severity: LOW

Found in the August 2026 session lifecycle review (#319).

The only backup mechanism in the product writes a pg_dump to a directory on the same host it's backing up, is disabled by default, and never touches the audio volume at all — so the single most catastrophic failure mode for a hosted "data is never lost" product (losing the host) takes the database, the audio, and the backups of the database with it, all at once.

Evidence

webapp/backend/app/tasks/reminder_tasks.py:3179-3200 (run_scheduled_backup/_run_backup_async) — runs pg_dump to a local directory on the same host (confirmed by the subprocess invocation and surrounding retention-pruning logic operating on local paths), is off by default (only runs when explicitly enabled in admin settings, or via the admin "Run now" force=True override), and prunes old local dumps on a retention schedule — nothing in this task, or anywhere else in reminder_tasks.py, backs up the audio_temp volume.

Failure scenario
The Docker host running a hosted Quest Board instance suffers a disk failure or is lost entirely (hardware fault, accidental deprovisioning, ransomware). The database, the audio recordings, and every local pg_dump backup that had been taken were all on that same host's storage — everything is gone simultaneously, for every tenant on that instance, with no off-host copy of anything to restore from.

Proposed fix
Ship pg_dump to off-host storage (object storage — S3-compatible — is the natural fit given the rest of the stack), enable it by default for the hosted profile specifically (self-hosters can opt out), and add an equivalent backup path for the audio volume. Periodically test a real restore (not just verify the dump completes) as part of the ops runbook, since an untested backup is not a verified one.

Acceptance criteria

run_scheduled_backup (or a new task) writes the database dump to off-host storage, not solely to local disk.
Off-host backups are enabled by default for the hosted deployment profile.
The audio volume has an equivalent backup path.
A documented, periodically-exercised restore procedure exists (docs/OPERATIONS.md).

**Severity: LOW** Found in the August 2026 session lifecycle review (#319). The only backup mechanism in the product writes a `pg_dump` to a directory on the same host it's backing up, is disabled by default, and never touches the audio volume at all — so the single most catastrophic failure mode for a hosted "data is never lost" product (losing the host) takes the database, the audio, and the backups of the database with it, all at once. **Evidence** - `webapp/backend/app/tasks/reminder_tasks.py:3179-3200` (`run_scheduled_backup`/`_run_backup_async`) — runs `pg_dump` to a local directory on the same host (confirmed by the `subprocess` invocation and surrounding retention-pruning logic operating on local paths), is off by default (only runs when explicitly enabled in admin settings, or via the admin "Run now" `force=True` override), and prunes old local dumps on a retention schedule — nothing in this task, or anywhere else in `reminder_tasks.py`, backs up the `audio_temp` volume. **Failure scenario** The Docker host running a hosted Quest Board instance suffers a disk failure or is lost entirely (hardware fault, accidental deprovisioning, ransomware). The database, the audio recordings, and every local pg_dump backup that had been taken were all on that same host's storage — everything is gone simultaneously, for every tenant on that instance, with no off-host copy of anything to restore from. **Proposed fix** Ship pg_dump to off-host storage (object storage — S3-compatible — is the natural fit given the rest of the stack), enable it by default for the hosted profile specifically (self-hosters can opt out), and add an equivalent backup path for the audio volume. Periodically test a real restore (not just verify the dump completes) as part of the ops runbook, since an untested backup is not a verified one. **Acceptance criteria** - [ ] `run_scheduled_backup` (or a new task) writes the database dump to off-host storage, not solely to local disk. - [ ] Off-host backups are enabled by default for the hosted deployment profile. - [ ] The audio volume has an equivalent backup path. - [ ] A documented, periodically-exercised restore procedure exists (`docs/OPERATIONS.md`).

claude-bot added this to the v4.1.0 - Data Durability & Recovery milestone

2026-08-25 20:44:55 +00:00

claude-bot added the

backend

enhancement

labels

2026-08-25 20:44:55 +00:00

claude-bot referenced this issue

2026-08-29 18:03:43 +00:00

[Frontend] First-run setup wizard so nothing gets forgotten on a fresh install #439

rbrooks referenced this issue from a commit

2026-08-29 18:32:56 +00:00

feat(ops): back up by default, somewhere the operator can see (#414)

claude-bot referenced this issue

2026-08-29 18:33:15 +00:00

feat(ops): back up by default, somewhere the operator can see (#414) #446

rbrooks referenced this issue from a commit

2026-08-29 19:18:36 +00:00

feat(ops): back up by default, somewhere the operator can see (#414)

claude-bot referenced this issue from a commit

2026-08-29 22:26:48 +00:00

Merge PR #446: back up by default, somewhere the operator can see (#414)

rbrooks referenced this issue from a commit

2026-08-31 22:09:11 +00:00

feat(ops): ship backups off-host and cover the audio volume (#414)

claude-bot referenced this issue

2026-08-31 22:09:49 +00:00

feat(ops): ship backups off-host and cover the audio volume (#414) #472

rbrooks referenced this issue from a commit

2026-08-31 22:21:29 +00:00

fix(security): stop a backup remote being read as rclone flags (#414)

claude-bot referenced this issue from a commit

2026-08-31 22:42:30 +00:00

Merge PR #472: ship backups off-host and cover the audio volume (#414)

claude-bot commented

2026-08-31 22:43:03 +00:00

Author

Collaborator

Merged in #472 (0a9055b). Closing.

Acceptance criteria

run_scheduled_backup writes the dump to off-host storage. An rclone destination in Admin → Backups; each artefact is copied there after the dump. A backup that cannot be shipped is recorded as failed, even though pg_dump succeeded — a dump on the disk it protects is the exact state this issue exists to stop reporting as a working backup. The local file is kept.
Enabled by default — but see the scope note below; this had already partly landed.
The audio volume has an equivalent backup path. Opt-in tar of audio_temp, shipped alongside the dump, with a restore procedure.
Documented restore procedure in docs/OPERATIONS.md — see the note on "periodically exercised" below.

Two deliberate scope calls

Criterion 2 as literally written presupposes something that does not exist. There is no deployment-profile concept anywhere in the codebase — no DEPLOYMENT_PROFILE, no hosted/self-hosted switch — so "enabled by default for the hosted deployment profile" had nothing to attach to. Off-host shipping activates when a remote is configured instead. Inventing a hosted/self-hosted profile is a product decision that would affect far more than backups, and is worth its own issue if wanted rather than arriving as a side effect of this one.

Note also that the other half of criterion 2 had already landed before this work started: scheduled backups have been on by default since an earlier pass on this same issue (get_backup_config's docstring cites #414), and the backups bind mount in docker-compose.yml cites it too. The issue body never reflected that.

Criterion 4's "periodically exercised" is an ops practice, not code. The runbook exists — and already said "A dump you have never restored is an untested backup" — and now covers audio as well. But nothing in a merged PR makes anyone actually rehearse a restore. If that should be tracked, it wants a recurring ops task rather than a closeable issue.

The most dangerous thing found here was in the docs

docs/OPERATIONS.md described the audio volume as:

deliberately excluded. It is transient scratch space … Nothing here needs to survive a restore.

Untrue for some time: process_audio never deletes a session directory (it waits for GM approval), retain_indefinitely is a supported per-campaign mode, trashed audio is restorable through a 7-day minimum grace (#427), and a session whose transcription failed has no transcript, so its audio is the only record it happened (#402). An operator following that advice would have lost every recording on the instance during a restore.

Worth noting how it happened: the sentence was true when written. Retention behaviour moved underneath it across several issues and the sentence did not move with it. That is the fifth or sixth time this milestone that an issue's or a doc's stated evidence had decayed — see also the line references in this issue's own body, which had drifted from 3179–3200 to 3942–4079.

Design decisions worth keeping

rclone copy, never rclone sync. sync mirrors, so a run whose local backup directory was empty — a fresh volume, a bad mount, the disk failure this feature is for — would delete the entire off-host set at the moment it became the only copy. Same shape as reconcile_audio_directories' empty-table guard from #406. The cost is that we never prune the remote, which is documented and is the right way round.
Audio snapshots are off by default, unlike backups themselves. Enabling dumps by default costs megabytes; enabling audio by default could multiply an existing install's storage on upgrade, and a full volume breaks the database backups too — turning a durability feature into the outage it prevents.
The remote is treated as a secret. rclone accepts inline connection strings carrying live credentials, which would otherwise reach backup_logs.destination, the audit log, and rclone's stderr. This retired the # No secrets in backup config comment in admin.py, which was true right up until that commit.

Follow-up already handled

An automated security review flagged argument injection: the remote's character class admitted a leading -, so --config:x and similar passed validation into subprocess.run's argv. Probing the rclone in our own image showed it was not exploitable — every such probe returns unknown flag, because the validator also demands a ,/: that pflag does not accept in flag syntax. Fixed anyway in 7d1fd5c, because that reasoning is a property of rclone's parser rather than of our input handling and would need re-deriving on every upgrade: validation now rejects a leading -, and ship_to_remote terminates flag parsing with -- before the operands.

Merged in #472 (`0a9055b`). Closing. ## Acceptance criteria - [x] **`run_scheduled_backup` writes the dump to off-host storage.** An rclone destination in Admin → Backups; each artefact is copied there after the dump. A backup that cannot be shipped is recorded as **failed**, even though `pg_dump` succeeded — a dump on the disk it protects is the exact state this issue exists to stop reporting as a working backup. The local file is kept. - [x] **Enabled by default** — but see the scope note below; this had already partly landed. - [x] **The audio volume has an equivalent backup path.** Opt-in tar of `audio_temp`, shipped alongside the dump, with a restore procedure. - [x] **Documented restore procedure** in `docs/OPERATIONS.md` — see the note on "periodically exercised" below. ## Two deliberate scope calls **Criterion 2 as literally written presupposes something that does not exist.** There is no deployment-profile concept anywhere in the codebase — no `DEPLOYMENT_PROFILE`, no hosted/self-hosted switch — so "enabled by default for the hosted deployment profile" had nothing to attach to. Off-host shipping activates when a remote is configured instead. Inventing a hosted/self-hosted profile is a product decision that would affect far more than backups, and is worth its own issue if wanted rather than arriving as a side effect of this one. Note also that the *other* half of criterion 2 had already landed before this work started: scheduled backups have been on by default since an earlier pass on this same issue (`get_backup_config`'s docstring cites #414), and the `backups` bind mount in `docker-compose.yml` cites it too. The issue body never reflected that. **Criterion 4's "periodically exercised" is an ops practice, not code.** The runbook exists — and already said "A dump you have never restored is an untested backup" — and now covers audio as well. But nothing in a merged PR makes anyone actually rehearse a restore. If that should be tracked, it wants a recurring ops task rather than a closeable issue. ## The most dangerous thing found here was in the docs `docs/OPERATIONS.md` described the audio volume as: > deliberately **excluded**. It is transient scratch space … Nothing here needs to survive a restore. Untrue for some time: `process_audio` never deletes a session directory (it waits for GM approval), `retain_indefinitely` is a supported per-campaign mode, trashed audio is restorable through a 7-day minimum grace (#427), and a session whose transcription **failed** has no transcript, so its audio is the only record it happened (#402). An operator following that advice would have lost every recording on the instance during a restore. Worth noting how it happened: the sentence was true when written. Retention behaviour moved underneath it across several issues and the sentence did not move with it. That is the fifth or sixth time this milestone that an issue's or a doc's stated evidence had decayed — see also the line references in this issue's own body, which had drifted from 3179–3200 to 3942–4079. ## Design decisions worth keeping - **`rclone copy`, never `rclone sync`.** `sync` mirrors, so a run whose local backup directory was empty — a fresh volume, a bad mount, the disk failure this feature is *for* — would delete the entire off-host set at the moment it became the only copy. Same shape as `reconcile_audio_directories`' empty-table guard from #406. The cost is that we never prune the remote, which is documented and is the right way round. - **Audio snapshots are off by default, unlike backups themselves.** Enabling dumps by default costs megabytes; enabling audio by default could multiply an existing install's storage on upgrade, and a full volume breaks the database backups too — turning a durability feature into the outage it prevents. - **The remote is treated as a secret.** rclone accepts inline connection strings carrying live credentials, which would otherwise reach `backup_logs.destination`, the audit log, and rclone's stderr. This retired the `# No secrets in backup config` comment in `admin.py`, which was true right up until that commit. ## Follow-up already handled An automated security review flagged argument injection: the remote's character class admitted a leading `-`, so `--config:x` and similar passed validation into `subprocess.run`'s argv. Probing the rclone in our own image showed it was **not** exploitable — every such probe returns `unknown flag`, because the validator also demands a `,`/`:` that pflag does not accept in flag syntax. Fixed anyway in `7d1fd5c`, because that reasoning is a property of rclone's parser rather than of our input handling and would need re-deriving on every upgrade: validation now rejects a leading `-`, and `ship_to_remote` terminates flag parsing with `--` before the operands.

claude-bot closed this issue

2026-08-31 22:43:03 +00:00

claude-bot referenced this issue

2026-09-22 14:42:19 +00:00

Build lane 2: the public pages (PUB-1 to PUB-12) #728

claude-bot referenced this issue

2026-09-22 16:11:50 +00:00

[Backend] A backup shipped off-host cannot be read back: its local path is overwritten by the rclone spec #753

rbrooks referenced this issue from a commit

2026-09-22 16:34:22 +00:00

feat(backend): first-run setup's own state, and a backup read-back (#728)

claude-bot referenced this issue