Off-site / object-storage backup target for the SQLite DB

claude-bot commented

2026-07-07 16:52:40 +00:00

Contributor

Goal

Optionally ship the existing timestamped SQLite backups to an off-box target (S3-compatible object storage, or a mounted remote path) so a host failure doesn't lose approved schemes and history.

Why it's valuable

v1 already creates timestamped backups before bulk operations and prunes by BACKUP_RETAIN_COUNT, but they live on the same volume as the DB. Off-site copies make the data actually durable — important because approved schemes represent real curation effort.

Sketch

Env-gated backup target (e.g. BACKUP_S3_ENDPOINT/bucket/creds, or BACKUP_REMOTE_PATH).
After each local backup, upload the artifact; apply the same retention remotely.
Restore path documented in docs/troubleshooting.md.

Acceptance criteria

Backups optionally uploaded to an S3-compatible or remote-path target
Remote retention honored
Feature env-gated, off by default; restore documented

Proposed enhancement (not in the original spec).

## Goal Optionally ship the existing timestamped SQLite backups to an off-box target (S3-compatible object storage, or a mounted remote path) so a host failure doesn't lose approved schemes and history. ## Why it's valuable v1 already creates timestamped backups before bulk operations and prunes by `BACKUP_RETAIN_COUNT`, but they live on the same volume as the DB. Off-site copies make the data actually durable — important because approved schemes represent real curation effort. ## Sketch - Env-gated backup target (e.g. `BACKUP_S3_ENDPOINT`/bucket/creds, or `BACKUP_REMOTE_PATH`). - After each local backup, upload the artifact; apply the same retention remotely. - Restore path documented in `docs/troubleshooting.md`. ## Acceptance criteria - [ ] Backups optionally uploaded to an S3-compatible or remote-path target - [ ] Remote retention honored - [ ] Feature env-gated, off by default; restore documented Proposed enhancement (not in the original spec).

claude-bot added the

labels

2026-07-07 16:53:43 +00:00

claude-bot referenced this issue

2026-07-07 17:38:09 +00:00

Declarative full-config export / import (GitOps) #23

claude-bot added this to the v1.1.0 milestone

2026-07-15 15:32:58 +00:00

claude-bot referenced this issue

2026-07-15 15:38:30 +00:00

Scheduled daily DB backup and a tested restore procedure #57

rbrooks referenced this issue from a commit

2026-09-04 23:01:33 +00:00

Mirror the SQLite backups to an off-site target (#18)

claude-bot referenced this issue from a pull request that will close it,

2026-09-04 23:01:49 +00:00

Mirror the SQLite backups to an off-site target (#18) #123

claude-bot closed this issue

2026-09-04 23:07:06 +00:00

claude-bot referenced this issue from a commit

2026-09-04 23:07:06 +00:00

Mirror the SQLite backups to an off-site target (#18)

claude-bot commented

2026-09-04 23:08:35 +00:00

Author

Contributor

Done — #123 merged, CI green.

Backups optionally uploaded to an S3-compatible or remote-path target
Remote retention honored
Feature env-gated, off by default; restore documented (docs/troubleshooting.md)

A real bug in the first draft

Remote retention originally matched on an "iris_" filename prefix. Every daily backup is named iris_daily_..., so that prefix matches the dailies too — manual pruning would have deleted daily backups early. It now shares the exact _KIND_GLOBS pattern local pruning uses, deliberately one rule rather than two that must be kept in step.

Why no boto3

S3 here is PUT, list and DELETE, against ~50MB of SDK in an image deliberately slimmed in #60. The SigV4 signature is computed on httpx directly.

That is normally a bad trade, and the reason it is acceptable here is specific: SigV4 is request authentication, not encryption. No key exchange, no confidentiality property to get subtly wrong — a mistake yields a 403 and a loudly broken feature rather than a silently insecure one.

defusedxml (~30KB) was added, for the listing response. Refusing that would have been false economy: the body is remote input and the stdlib parser expands entities, so a hostile or compromised endpoint could answer a listing with a billion-laughs payload. The response is size-bounded as well.

On the signing test

Worth recording because I got it wrong twice. I first asserted a hex vector recalled from memory; it failed. I "fixed" it by matching AWS's iam example; it failed again, because the constant itself was fabricated. Fetching AWS's actual documentation settled it: the current docs state the derivation as an algorithm and give no worked numeric example, so there is no vendor vector to cite.

The implemented chain matches the spec exactly (AWS4+secret → date → region → service → aws4_request). The test now reimplements those four steps longhand from the specification rather than calling the module and comparing it to itself, and I confirmed it catches a region/service swap. I also removed the service parameter that existed only to serve the bogus test.

Better to say plainly that there is no vendor vector than to leave a comment claiming an authority the test does not have.

Failure behaviour

An unreachable target never fails a backup — the local snapshot is already written and is what actually stands between the user and data loss, so a dead NAS is logged and the operation that triggered it still completes. Documented in troubleshooting, along with the corollary: check the logs rather than assuming silence means success.

Uploads to a mounted path stage under a dot-prefixed name and are renamed, so a reader never sees a half-copied backup that looks complete.

Done — #123 merged, CI green. - [x] **Backups optionally uploaded to an S3-compatible or remote-path target** - [x] **Remote retention honored** - [x] **Feature env-gated, off by default; restore documented** (`docs/troubleshooting.md`) ### A real bug in the first draft Remote retention originally matched on an `"iris_"` filename prefix. Every daily backup is named `iris_daily_...`, so that prefix matches the dailies too — manual pruning would have deleted daily backups early. It now shares the exact `_KIND_GLOBS` pattern local pruning uses, deliberately one rule rather than two that must be kept in step. ### Why no boto3 S3 here is PUT, list and DELETE, against ~50MB of SDK in an image deliberately slimmed in #60. The SigV4 signature is computed on httpx directly. That is normally a bad trade, and the reason it is acceptable here is specific: **SigV4 is request authentication, not encryption.** No key exchange, no confidentiality property to get subtly wrong — a mistake yields a 403 and a loudly broken feature rather than a silently insecure one. `defusedxml` (~30KB) *was* added, for the listing response. Refusing that would have been false economy: the body is remote input and the stdlib parser expands entities, so a hostile or compromised endpoint could answer a listing with a billion-laughs payload. The response is size-bounded as well. ### On the signing test Worth recording because I got it wrong twice. I first asserted a hex vector recalled from memory; it failed. I "fixed" it by matching AWS's `iam` example; it failed again, because the constant itself was fabricated. Fetching AWS's actual documentation settled it: the current docs state the derivation as an algorithm and give **no worked numeric example**, so there is no vendor vector to cite. The implemented chain matches the spec exactly (`AWS4+secret → date → region → service → aws4_request`). The test now reimplements those four steps longhand from the specification rather than calling the module and comparing it to itself, and I confirmed it catches a region/service swap. I also removed the `service` parameter that existed only to serve the bogus test. Better to say plainly that there is no vendor vector than to leave a comment claiming an authority the test does not have. ### Failure behaviour An unreachable target never fails a backup — the local snapshot is already written and is what actually stands between the user and data loss, so a dead NAS is logged and the operation that triggered it still completes. Documented in troubleshooting, along with the corollary: check the logs rather than assuming silence means success. Uploads to a mounted path stage under a dot-prefixed name and are renamed, so a reader never sees a half-copied backup that looks complete.

Rows
Columns

Off-site / object-storage backup target for the SQLite DB #18