Implement the S3 storage backend #41

New issue

Open

opened 2026-07-28 04:59:52 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-28 04:59:52 +00:00

Context

Storage is already behind an abstraction with a local filesystem implementation. The
Phase 4 acceptance criterion is that swapping storage backends requires no core
application logic change â€” this issue is the test of whether that abstraction actually holds.

Scope

An S3-compatible storage backend selectable by configuration.

Implementation notes

Implement against the existing storage interface. If the interface needs changes to
accommodate object storage, that is a finding worth noting â€” but core services must not
learn anything about S3.
Target S3-compatible APIs generally (MinIO, Backblaze B2, Wasabi), not AWS specifically.
Keep content-addressing by SHA-256 as the key scheme, consistent with local storage.
Media serving should use presigned URLs rather than proxying image bytes through the
backend â€” but presigned URLs bypass the app's authorization checks, so their expiry must
be short and the generation path must verify the requesting user may see that photo.
Handle the failure modes local storage does not have: network errors, partial uploads,
eventual consistency, and throttling. Retry with backoff.
Multipart upload for large files.
Credentials from environment or an instance role, never from the repo.
A migration path for moving an existing local collection to S3, with verification that
every object arrived intact before anything local is removed.

Done when

S3-compatible storage is selectable by configuration alone
No core service contains storage-backend-specific logic
Media serving works with short-lived presigned URLs that respect authorization
Network failures are retried and surfaced rather than silently dropped
Local-to-S3 migration is verified before any local deletion

References

backend/app/services/storage.py
docs/circa-spec.md Phase 4 acceptance criteria

## Context Storage is already behind an abstraction with a local filesystem implementation. The Phase 4 acceptance criterion is that swapping storage backends requires no core application logic change â€” this issue is the test of whether that abstraction actually holds. ## Scope An S3-compatible storage backend selectable by configuration. ## Implementation notes - Implement against the existing storage interface. If the interface needs changes to accommodate object storage, that is a finding worth noting â€” but core services must not learn anything about S3. - Target S3-compatible APIs generally (MinIO, Backblaze B2, Wasabi), not AWS specifically. - Keep content-addressing by SHA-256 as the key scheme, consistent with local storage. - Media serving should use presigned URLs rather than proxying image bytes through the backend â€” but presigned URLs bypass the app's authorization checks, so their expiry must be short and the generation path must verify the requesting user may see that photo. - Handle the failure modes local storage does not have: network errors, partial uploads, eventual consistency, and throttling. Retry with backoff. - Multipart upload for large files. - Credentials from environment or an instance role, never from the repo. - A migration path for moving an existing local collection to S3, with verification that every object arrived intact before anything local is removed. ## Done when - [ ] S3-compatible storage is selectable by configuration alone - [ ] No core service contains storage-backend-specific logic - [ ] Media serving works with short-lived presigned URLs that respect authorization - [ ] Network failures are retried and surfaced rather than silently dropped - [ ] Local-to-S3 migration is verified before any local deletion ## References - `backend/app/services/storage.py` - `docs/circa-spec.md` Phase 4 acceptance criteria

claude-bot added this to the v0.6.0 milestone

2026-07-28 04:59:52 +00:00

claude-bot added the

backend

enhancement

labels

2026-07-28 04:59:52 +00:00

claude-bot referenced this issue

2026-07-28 06:00:31 +00:00

Storage read interface returns a Path and cannot support S3 #90

claude-bot referenced this issue

2026-07-28 06:06:08 +00:00

Assorted spec items: quality flagging, search, rebuild job, storage ops #119

claude-bot commented

2026-07-28 06:07:52 +00:00

Author

Amended by the audit of 2026-07-28.

This issue will collide with the storage interface as designed. StorageBackend.get_path(key) -> Path means "return a filesystem path the caller reads directly" (consumed by FileResponse), and
S3 cannot honour that without downloading every object to a temp file.

So the v0.6.0 exit criterion — "swapping storage backends requires no core application logic
change" — fails at the interface, not the implementation.

#90 changes the read contract to open(key) -> BinaryIO plus an optional public_url(key) for
presigned redirects, while there is one implementation and two call sites. It should land well
before this issue.

Note also that presigned URLs bypass the app's authorization checks, so their expiry must be short
and the generating path must verify the requester may see that photo.

**Amended by the audit of 2026-07-28.** This issue will collide with the storage interface as designed. `StorageBackend.get_path(key) -> Path` means "return a filesystem path the caller reads directly" (consumed by `FileResponse`), and S3 cannot honour that without downloading every object to a temp file. So the v0.6.0 exit criterion — "swapping storage backends requires no core application logic change" — fails at the *interface*, not the implementation. #90 changes the read contract to `open(key) -> BinaryIO` plus an optional `public_url(key)` for presigned redirects, while there is one implementation and two call sites. It should land well before this issue. Note also that presigned URLs bypass the app's authorization checks, so their expiry must be short and the generating path must verify the requester may see that photo.