Implement near-duplicate detection via perceptual hashing #16

New issue

Open

opened 2026-07-28 04:55:02 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-28 04:55:02 +00:00

Context

v0.1.0 detects exact duplicates by SHA-256, but a photo scanned twice â€” different
scanner settings, a re-scan after a crooked first pass, or the same print appearing in
two albums â€” produces different bytes and slips through. The schema already anticipates
this: DuplicateType.near and JobType.duplicate_scan exist and are unused.

Scope

Perceptual-hash computation at ingest, a backfill job for the existing collection, and
candidate near-duplicate pair detection.

Implementation notes

Compute a perceptual hash (pHash, or dHash for speed) at ingest alongside the existing
SHA-256, and store it on Photo. Requires a migration.
Comparing every photo against every other is quadratic and will not hold for a large
family collection. Bucket candidates first â€” hamming-distance-indexed prefixes or a
BK-tree â€” and document the chosen approach and its expected cost.
The duplicate_scan job type handles backfill over already-ingested photos so the
feature applies retroactively, not just to new scans.
Store detected pairs with their distance score so the review UI can order by confidence.
The distance threshold must be configurable and should start conservative. A false
positive shown to a reviewer is cheap; a false negative that silently merges two
distinct photographs is not â€” and crucially, nothing here may auto-merge or delete.
Detection proposes; the reviewer disposes.
Photos legitimately similar but distinct (a burst of near-identical portraits, two prints
of one negative) must remain independently reviewable.

Done when

Perceptual hashes are computed at ingest and stored
A backfill job hashes the existing collection
Near-duplicate candidate pairs are detected with a distance score
Detection never modifies or removes a photo on its own
The approach is documented with its scaling characteristics

References

backend/app/models/models.py (DuplicateType.near, JobType.duplicate_scan)
backend/app/services/ingest.py (existing SHA-256 exact detection)
docs/circa-spec.md Phase 2

Depends on: #2 (worker runtime).

## Context v0.1.0 detects exact duplicates by SHA-256, but a photo scanned twice â€” different scanner settings, a re-scan after a crooked first pass, or the same print appearing in two albums â€” produces different bytes and slips through. The schema already anticipates this: `DuplicateType.near` and `JobType.duplicate_scan` exist and are unused. ## Scope Perceptual-hash computation at ingest, a backfill job for the existing collection, and candidate near-duplicate pair detection. ## Implementation notes - Compute a perceptual hash (pHash, or dHash for speed) at ingest alongside the existing SHA-256, and store it on `Photo`. Requires a migration. - Comparing every photo against every other is quadratic and will not hold for a large family collection. Bucket candidates first â€” hamming-distance-indexed prefixes or a BK-tree â€” and document the chosen approach and its expected cost. - The `duplicate_scan` job type handles backfill over already-ingested photos so the feature applies retroactively, not just to new scans. - Store detected pairs with their distance score so the review UI can order by confidence. - The distance threshold must be configurable and should start conservative. A false positive shown to a reviewer is cheap; a false negative that silently merges two distinct photographs is not â€” and crucially, **nothing here may auto-merge or delete**. Detection proposes; the reviewer disposes. - Photos legitimately similar but distinct (a burst of near-identical portraits, two prints of one negative) must remain independently reviewable. ## Done when - [ ] Perceptual hashes are computed at ingest and stored - [ ] A backfill job hashes the existing collection - [ ] Near-duplicate candidate pairs are detected with a distance score - [ ] Detection never modifies or removes a photo on its own - [ ] The approach is documented with its scaling characteristics ## References - `backend/app/models/models.py` (`DuplicateType.near`, `JobType.duplicate_scan`) - `backend/app/services/ingest.py` (existing SHA-256 exact detection) - `docs/circa-spec.md` Phase 2 Depends on: #2 (worker runtime).

claude-bot added this to the v0.4.0 milestone

2026-07-28 04:55:02 +00:00

claude-bot added the

enhancement

worker

backend

labels

2026-07-28 04:55:02 +00:00

claude-bot referenced this issue

2026-07-28 06:06:03 +00:00

Group/stack decisions: date the event, not the photo #109

claude-bot commented

2026-08-05 19:38:39 +00:00

Author

Context from #144, which deliberately left this alone.

#144 wired ingest up to enqueue OCR, and stopped there. docs/circa-spec.md §8.3 step 9 also asks ingest to enqueue near-duplicate detection — this issue — and that half is still unwired: JobType.duplicate_scan remains unused, exactly as the body says. Nothing has changed about that; recording it so the gap is not rediscovered as a surprise.

Three things landed since this was written that change how it should be built.

1. Ingest is no longer one path, and "compute a perceptual hash at ingest" now has to say where. #105 gave ingest_photo five outcomes. Two of them matter here:

a fold writes a front onto a row that already existed as an orphan back. The row gets a front scan for the first time without going through the ordinary insert — so a pHash computed only on the insert path would leave every folded photograph unhashed, and folds are the normal case for a collection scanned over more than one session (§8.2).
an orphan back has no front scan at all. There is nothing to perceptually hash, because the only image is the reverse of a print. Whatever this stores must be able to say "not hashed, and correctly so" rather than treating it as a gap to backfill.

The lesson #105 paid for is worth reusing: derive the condition from the variable that decided where the bytes went, not from a list of outcomes. services/ingest.py does this for the OCR enqueue and the comment there explains why.

2. There is now one place to enqueue from. app/services/jobs.py::enqueue_photo_job holds the attempt numbering, the idempotency key, the in-flight fast path and the race recovery, and is called by both the rerun route and ingest. Use it rather than writing a third copy — two restatements of one rule is what caused a real bug in #105.

Note its IntegrityError recovery is a savepoint, not db.rollback(), specifically so a queue collision cannot discard an in-flight ingest's photograph.

3. sha256_back exists now, added in #105 so a re-presented back scan can recognise itself. Not a perceptual hash and no substitute for one, but worth knowing before adding another hash column: the ingest ledger (ingest_record) also records the content hash of every file ever presented, which may be a cheaper place to answer some of the backfill questions than a pass over originals.

Nothing here changes the scope or the conclusions in the body — the quadratic-comparison warning and the "detection proposes, the reviewer disposes" rule are untouched and remain the important parts.

Context from #144, which deliberately left this alone. #144 wired ingest up to enqueue OCR, and stopped there. `docs/circa-spec.md` §8.3 step 9 also asks ingest to enqueue **near-duplicate detection** — this issue — and that half is still unwired: `JobType.duplicate_scan` remains unused, exactly as the body says. Nothing has changed about that; recording it so the gap is not rediscovered as a surprise. Three things landed since this was written that change how it should be built. **1. Ingest is no longer one path, and "compute a perceptual hash at ingest" now has to say *where*.** #105 gave `ingest_photo` five outcomes. Two of them matter here: - a **fold** writes a front onto a row that already existed as an orphan back. The row gets a front scan for the first time *without* going through the ordinary insert — so a pHash computed only on the insert path would leave every folded photograph unhashed, and folds are the normal case for a collection scanned over more than one session (§8.2). - an **orphan back** has no front scan at all. There is nothing to perceptually hash, because the only image is the reverse of a print. Whatever this stores must be able to say "not hashed, and correctly so" rather than treating it as a gap to backfill. The lesson #105 paid for is worth reusing: derive the condition from the variable that decided where the bytes went, not from a list of outcomes. `services/ingest.py` does this for the OCR enqueue and the comment there explains why. **2. There is now one place to enqueue from.** `app/services/jobs.py::enqueue_photo_job` holds the attempt numbering, the idempotency key, the in-flight fast path and the race recovery, and is called by both the rerun route and ingest. Use it rather than writing a third copy — two restatements of one rule is what caused a real bug in #105. Note its `IntegrityError` recovery is a **savepoint**, not `db.rollback()`, specifically so a queue collision cannot discard an in-flight ingest's photograph. **3. `sha256_back` exists now**, added in #105 so a re-presented back scan can recognise itself. Not a perceptual hash and no substitute for one, but worth knowing before adding another hash column: the ingest ledger (`ingest_record`) also records the content hash of every file ever presented, which may be a cheaper place to answer some of the backfill questions than a pass over originals. Nothing here changes the scope or the conclusions in the body — the quadratic-comparison warning and the "detection proposes, the reviewer disposes" rule are untouched and remain the important parts.