[Ops] Move backups off-host and cover the audio volume #414
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Severity: LOW
Found in the August 2026 session lifecycle review (#319).
The only backup mechanism in the product writes a
pg_dumpto a directory on the same host it's backing up, is disabled by default, and never touches the audio volume at all — so the single most catastrophic failure mode for a hosted "data is never lost" product (losing the host) takes the database, the audio, and the backups of the database with it, all at once.Evidence
webapp/backend/app/tasks/reminder_tasks.py:3179-3200(run_scheduled_backup/_run_backup_async) — runspg_dumpto a local directory on the same host (confirmed by thesubprocessinvocation and surrounding retention-pruning logic operating on local paths), is off by default (only runs when explicitly enabled in admin settings, or via the admin "Run now"force=Trueoverride), and prunes old local dumps on a retention schedule — nothing in this task, or anywhere else inreminder_tasks.py, backs up theaudio_tempvolume.Failure scenario
The Docker host running a hosted Quest Board instance suffers a disk failure or is lost entirely (hardware fault, accidental deprovisioning, ransomware). The database, the audio recordings, and every local pg_dump backup that had been taken were all on that same host's storage — everything is gone simultaneously, for every tenant on that instance, with no off-host copy of anything to restore from.
Proposed fix
Ship pg_dump to off-host storage (object storage — S3-compatible — is the natural fit given the rest of the stack), enable it by default for the hosted profile specifically (self-hosters can opt out), and add an equivalent backup path for the audio volume. Periodically test a real restore (not just verify the dump completes) as part of the ops runbook, since an untested backup is not a verified one.
Acceptance criteria
run_scheduled_backup(or a new task) writes the database dump to off-host storage, not solely to local disk.docs/OPERATIONS.md).Merged in #472 (
0a9055b). Closing.Acceptance criteria
run_scheduled_backupwrites the dump to off-host storage. An rclone destination in Admin → Backups; each artefact is copied there after the dump. A backup that cannot be shipped is recorded as failed, even thoughpg_dumpsucceeded — a dump on the disk it protects is the exact state this issue exists to stop reporting as a working backup. The local file is kept.audio_temp, shipped alongside the dump, with a restore procedure.docs/OPERATIONS.md— see the note on "periodically exercised" below.Two deliberate scope calls
Criterion 2 as literally written presupposes something that does not exist. There is no deployment-profile concept anywhere in the codebase — no
DEPLOYMENT_PROFILE, no hosted/self-hosted switch — so "enabled by default for the hosted deployment profile" had nothing to attach to. Off-host shipping activates when a remote is configured instead. Inventing a hosted/self-hosted profile is a product decision that would affect far more than backups, and is worth its own issue if wanted rather than arriving as a side effect of this one.Note also that the other half of criterion 2 had already landed before this work started: scheduled backups have been on by default since an earlier pass on this same issue (
get_backup_config's docstring cites #414), and thebackupsbind mount indocker-compose.ymlcites it too. The issue body never reflected that.Criterion 4's "periodically exercised" is an ops practice, not code. The runbook exists — and already said "A dump you have never restored is an untested backup" — and now covers audio as well. But nothing in a merged PR makes anyone actually rehearse a restore. If that should be tracked, it wants a recurring ops task rather than a closeable issue.
The most dangerous thing found here was in the docs
docs/OPERATIONS.mddescribed the audio volume as:Untrue for some time:
process_audionever deletes a session directory (it waits for GM approval),retain_indefinitelyis a supported per-campaign mode, trashed audio is restorable through a 7-day minimum grace (#427), and a session whose transcription failed has no transcript, so its audio is the only record it happened (#402). An operator following that advice would have lost every recording on the instance during a restore.Worth noting how it happened: the sentence was true when written. Retention behaviour moved underneath it across several issues and the sentence did not move with it. That is the fifth or sixth time this milestone that an issue's or a doc's stated evidence had decayed — see also the line references in this issue's own body, which had drifted from 3179–3200 to 3942–4079.
Design decisions worth keeping
rclone copy, neverrclone sync.syncmirrors, so a run whose local backup directory was empty — a fresh volume, a bad mount, the disk failure this feature is for — would delete the entire off-host set at the moment it became the only copy. Same shape asreconcile_audio_directories' empty-table guard from #406. The cost is that we never prune the remote, which is documented and is the right way round.backup_logs.destination, the audit log, and rclone's stderr. This retired the# No secrets in backup configcomment inadmin.py, which was true right up until that commit.Follow-up already handled
An automated security review flagged argument injection: the remote's character class admitted a leading
-, so--config:xand similar passed validation intosubprocess.run's argv. Probing the rclone in our own image showed it was not exploitable — every such probe returnsunknown flag, because the validator also demands a,/:that pflag does not accept in flag syntax. Fixed anyway in7d1fd5c, because that reasoning is a property of rclone's parser rather than of our input handling and would need re-deriving on every upgrade: validation now rejects a leading-, andship_to_remoteterminates flag parsing with--before the operands.