Add ingest, filename-parsing, EXIF, and duplicate-detection tests #7

New issue

Closed

opened 2026-07-28 04:52:50 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-28 04:52:50 +00:00

Context

Ingest is the entry point for every photo in the system and has the most parsing
surface area: filename conventions, EXIF quirks, and duplicate detection. None of
it is covered by tests today.

Scope

Tests for app/services/ingest.py, filename_parser.py, exif_extractor.py, and
SHA-256 duplicate detection.

Implementation notes

Filename parser: cover every case the spec documents â€” album slug, sequence number,
partial dates (198X, 1987-06), front/back indicators â€” plus the cases that should
not parse. Malformed filenames must degrade to "no evidence" rather than raising or
inventing a date.
EXIF: DateTimeOriginal, DateTimeDigitized, and DateTime precedence; missing EXIF;
corrupt EXIF blocks; and a scanner-set timestamp (the date the scan was made, not the
date the photo was taken) â€” this last case matters because a scan date is misleading
evidence if trusted at face value.
Duplicate detection: ingesting the same bytes twice must flag the duplicate and
destroy neither record. Assert that both rows survive and are linked.
Same image content under two different filenames, and different content under the
same filename, are distinct cases â€” cover both.
Ingest emits photo_ingested audit events and creates evidence rows for each
detected signal.
Storage is content-addressed by SHA-256: confirm the stored path derives from the
hash and that re-ingest does not write a second copy of identical bytes.

Done when

Every filename case in the spec has a test, including the negative cases
Missing and corrupt EXIF are handled without raising
Exact-duplicate ingest is flagged and non-destructive
A batch ingest of mixed valid/invalid files completes and reports per-file outcomes

References

backend/app/services/ingest.py, filename_parser.py, exif_extractor.py, storage.py
backend/app/api/routes/ingest.py
docs/circa-spec.md (filename conventions)

Depends on: the backend test harness.

## Context Ingest is the entry point for every photo in the system and has the most parsing surface area: filename conventions, EXIF quirks, and duplicate detection. None of it is covered by tests today. ## Scope Tests for `app/services/ingest.py`, `filename_parser.py`, `exif_extractor.py`, and SHA-256 duplicate detection. ## Implementation notes - Filename parser: cover every case the spec documents â€” album slug, sequence number, partial dates (`198X`, `1987-06`), front/back indicators â€” plus the cases that should *not* parse. Malformed filenames must degrade to "no evidence" rather than raising or inventing a date. - EXIF: `DateTimeOriginal`, `DateTimeDigitized`, and `DateTime` precedence; missing EXIF; corrupt EXIF blocks; and a scanner-set timestamp (the date the scan was made, not the date the photo was taken) â€” this last case matters because a scan date is misleading evidence if trusted at face value. - Duplicate detection: ingesting the same bytes twice must flag the duplicate and destroy neither record. Assert that both rows survive and are linked. - Same image content under two different filenames, and different content under the same filename, are distinct cases â€” cover both. - Ingest emits `photo_ingested` audit events and creates evidence rows for each detected signal. - Storage is content-addressed by SHA-256: confirm the stored path derives from the hash and that re-ingest does not write a second copy of identical bytes. ## Done when - [ ] Every filename case in the spec has a test, including the negative cases - [ ] Missing and corrupt EXIF are handled without raising - [ ] Exact-duplicate ingest is flagged and non-destructive - [ ] A batch ingest of mixed valid/invalid files completes and reports per-file outcomes ## References - `backend/app/services/ingest.py`, `filename_parser.py`, `exif_extractor.py`, `storage.py` - `backend/app/api/routes/ingest.py` - `docs/circa-spec.md` (filename conventions) Depends on: the backend test harness.

claude-bot added this to the v0.2.0 milestone

2026-07-28 04:52:50 +00:00

claude-bot added the

testing

backend

labels

2026-07-28 04:52:50 +00:00

rbrooks referenced this issue from a commit

2026-08-02 06:24:23 +00:00

Stop the filename parser inventing dates

rbrooks closed this issue

2026-08-02 06:24:23 +00:00

claude-bot commented

2026-08-02 06:25:06 +00:00

Author

Done in a6c2e9f.

What was actually missing

filename_parser.py had no tests at all. exif_extractor.py and duplicate detection were largely covered already — by test_exif_trust.py (tag precedence, scanner timestamps), test_exif_failure_visibility.py (missing, corrupt, unopenable), and test_ingest_integrity.py (exact-duplicate ingest, content-addressed keys, the canonical-row race). So the work concentrated where the coverage was zero, plus the two named cases nothing reached.

The parser had four defects

The issue's rule — malformed filenames degrade to "no evidence" rather than raising or inventing a date — held in three of the four ways it can fail. Writing the cases found:

An impossible date left .year / .month / .day populated with no range at all. A caller reading those fields built evidence out of a date the parser had itself rejected. This is the "inventing" failure, arrived at by leaving half a result behind.
3999-01-01 parsed but 3999 did not. The dashed patterns accepted any four digits while the bare-year pattern required 19xx/20xx.
0000-01-01 left a month and day behind after the year was refused.
A year was read as a sequence number on any name of the form <album>_<year> — a four-digit year matches _SEQ_PATTERN perfectly well. Photo "1983" of the album, sorting after 400 and before 2000. Wrong on every such name, in an ordering nobody would think to check.

None of these were cosmetic. _infer_precision in ingest.py reads .day and .month, so 3999-01-01 produced day-precision filename evidence at medium reliability — and medium is what promotes a photo to needs_review under projections.fold(). The tests now follow a name through ingest to the evidence row rather than stopping at the parser.

The fix

_plausible_year() — one bound for every pattern, shared with EXIF trust (#74), so the archive has one answer to "could this be a date in this collection".
_resolve() — degrades to the coarsest reading that is actually true, and never leaves partial state. 1983-02-30 is not the thirtieth of February, but it is still evidence of February 1983; discarding the whole match would lose a legible date over an illegible digit.
parse_filename() extracts the date first, records its span, and looks for the sequence outside it.

The invariant is asserted directly for every shape: no component outlives the range that supports it — which is what makes .year safe to read without also checking .date_low.

One of my own tests was wrong and got corrected rather than the code: I had asserted an impossible date should yield nothing at all, which throws away a true coarser reading.

Ingest cases added

Different content under the same filename stays two photographs. The false-positive direction of duplicate detection, and the expensive one: duplicate_of is what the review queue filters on, so a batch of scans all named scan.jpg — what an unattended scanner produces — would collapse to one visible photograph with the rest recorded as redundant. Nothing raises; they are simply gone from the queue.
The stored key derives from the digest, not the client's filename.
Mixed valid/invalid batch. There is no batch endpoint in Phase 1 (bulk ingest is v0.3.0), so this covers the property that makes one possible: each ingest stands alone, one file's outcome does not affect the next, and a rejected file leaves no row behind.

Done when

Every filename case in the spec has a test, including the negative cases
Missing and corrupt EXIF are handled without raising
Exact-duplicate ingest is flagged and non-destructive
A batch ingest of mixed valid/invalid files completes and reports per-file outcomes

787 passed, 8 skipped; ruff clean.

Done in a6c2e9f. ## What was actually missing `filename_parser.py` had **no tests at all**. `exif_extractor.py` and duplicate detection were largely covered already — by `test_exif_trust.py` (tag precedence, scanner timestamps), `test_exif_failure_visibility.py` (missing, corrupt, unopenable), and `test_ingest_integrity.py` (exact-duplicate ingest, content-addressed keys, the canonical-row race). So the work concentrated where the coverage was zero, plus the two named cases nothing reached. ## The parser had four defects The issue's rule — *malformed filenames degrade to "no evidence" rather than raising or inventing a date* — held in three of the four ways it can fail. Writing the cases found: 1. **An impossible date left `.year` / `.month` / `.day` populated with no range at all.** A caller reading those fields built evidence out of a date the parser had itself rejected. This is the "inventing" failure, arrived at by leaving half a result behind. 2. **`3999-01-01` parsed but `3999` did not.** The dashed patterns accepted any four digits while the bare-year pattern required `19xx`/`20xx`. 3. **`0000-01-01` left a month and day behind** after the year was refused. 4. **A year was read as a sequence number** on any name of the form `<album>_<year>` — a four-digit year matches `_SEQ_PATTERN` perfectly well. Photo "1983" of the album, sorting after 400 and before 2000. Wrong on every such name, in an ordering nobody would think to check. **None of these were cosmetic.** `_infer_precision` in `ingest.py` reads `.day` and `.month`, so `3999-01-01` produced *day-precision* filename evidence at `medium` reliability — and `medium` is what promotes a photo to `needs_review` under `projections.fold()`. The tests now follow a name through ingest to the evidence row rather than stopping at the parser. ## The fix - `_plausible_year()` — one bound for every pattern, shared with EXIF trust (#74), so the archive has one answer to "could this be a date in this collection". - `_resolve()` — degrades to the coarsest reading that is actually true, and never leaves partial state. `1983-02-30` is not the thirtieth of February, but it is still evidence of February 1983; discarding the whole match would lose a legible date over an illegible digit. - `parse_filename()` extracts the date **first**, records its span, and looks for the sequence outside it. The invariant is asserted directly for every shape: **no component outlives the range that supports it** — which is what makes `.year` safe to read without also checking `.date_low`. One of my own tests was wrong and got corrected rather than the code: I had asserted an impossible date should yield nothing at all, which throws away a true coarser reading. ## Ingest cases added - **Different content under the same filename stays two photographs.** The false-positive direction of duplicate detection, and the expensive one: `duplicate_of` is what the review queue filters on, so a batch of scans all named `scan.jpg` — what an unattended scanner produces — would collapse to one visible photograph with the rest recorded as redundant. Nothing raises; they are simply gone from the queue. - **The stored key derives from the digest**, not the client's filename. - **Mixed valid/invalid batch.** There is no batch endpoint in Phase 1 (bulk ingest is v0.3.0), so this covers the property that makes one possible: each ingest stands alone, one file's outcome does not affect the next, and a rejected file leaves no row behind. ## Done when - [x] Every filename case in the spec has a test, including the negative cases - [x] Missing and corrupt EXIF are handled without raising - [x] Exact-duplicate ingest is flagged and non-destructive - [x] A batch ingest of mixed valid/invalid files completes and reports per-file outcomes **787 passed, 8 skipped**; ruff clean.

claude-bot referenced this issue

2026-08-03 15:51:00 +00:00

Add an end-to-end reviewer flow test #12