Implement the periodic file integrity checker #47

New issue

Open

opened 2026-07-28 04:59:56 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-07-28 04:59:56 +00:00

Context

Storage is content-addressed by SHA-256, so every stored file's hash is already known.
Bit rot, a failing disk, or a bad sync can corrupt a scan silently, and the corruption
would only be discovered at export â€” possibly after the physical original has been
discarded. Periodic verification turns silent corruption into a detected event.

Scope

A background job that periodically re-verifies stored file hashes.

Implementation notes

Add an integrity check job type and run it on a schedule via the worker.
Re-hash stored files and compare against the recorded SHA-256.
Do not attempt to check the whole collection in one run. Work through it incrementally
with a persisted cursor, so a large collection is covered over time without saturating
disk I/O and making the app unusable while it runs.
Throttle deliberately; this is a background hygiene task and must yield to real work.
A mismatch is a serious event: record it, alert prominently through the notification
system, and mark the photo so it cannot be silently exported as if intact.
Record missing files distinctly from corrupt ones â€” they have different causes and
different remedies.
Report coverage, so it is clear how much of the collection has been verified and when.
Never auto-repair or auto-delete. Surface the problem and let a human decide.

Done when

Stored files are periodically re-hashed and compared
Verification is incremental, resumable, and throttled
Mismatches and missing files are recorded distinctly and alerted
Affected photos are flagged and excluded from clean export
Coverage and last-verified times are reportable
No automatic repair or deletion occurs

References

backend/app/services/storage.py (content-addressed by SHA-256)
docs/circa-spec.md Phase 4

Depends on: #2 (worker runtime), #27 (notifications).

## Context Storage is content-addressed by SHA-256, so every stored file's hash is already known. Bit rot, a failing disk, or a bad sync can corrupt a scan silently, and the corruption would only be discovered at export â€” possibly after the physical original has been discarded. Periodic verification turns silent corruption into a detected event. ## Scope A background job that periodically re-verifies stored file hashes. ## Implementation notes - Add an integrity check job type and run it on a schedule via the worker. - Re-hash stored files and compare against the recorded SHA-256. - Do not attempt to check the whole collection in one run. Work through it incrementally with a persisted cursor, so a large collection is covered over time without saturating disk I/O and making the app unusable while it runs. - Throttle deliberately; this is a background hygiene task and must yield to real work. - A mismatch is a serious event: record it, alert prominently through the notification system, and mark the photo so it cannot be silently exported as if intact. - Record missing files distinctly from corrupt ones â€” they have different causes and different remedies. - Report coverage, so it is clear how much of the collection has been verified and when. - Never auto-repair or auto-delete. Surface the problem and let a human decide. ## Done when - [ ] Stored files are periodically re-hashed and compared - [ ] Verification is incremental, resumable, and throttled - [ ] Mismatches and missing files are recorded distinctly and alerted - [ ] Affected photos are flagged and excluded from clean export - [ ] Coverage and last-verified times are reportable - [ ] No automatic repair or deletion occurs ## References - `backend/app/services/storage.py` (content-addressed by SHA-256) - `docs/circa-spec.md` Phase 4 Depends on: #2 (worker runtime), #27 (notifications).

claude-bot added this to the v0.6.0 milestone

2026-07-28 04:59:56 +00:00

claude-bot added the

worker

backend

labels

2026-07-28 04:59:56 +00:00

claude-bot referenced this issue

2026-07-28 06:00:32 +00:00

Ingest partial-write orphans, missing unique constraint, naive timestamps #91