rbrooks/TeaLeaves

Watch

Fork 0

Image dedup (spec Pass 3) doesn't exist, and `media.phash` is a SHA-256 of the bytes, not a perceptual hash #136

New issue

Closed

opened 2026-07-17 03:00:03 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-17 03:00:03 +00:00

Contributor

Severity: Low · Confidence: High · Effort: M · Category: code (+ docs drift)

Problem

Same class as #99 — a documented dedup pass that isn't implemented. Two separate issues:

1. There is no image dedup. The spec (Tea Leaves - spec.md:240) lists it as the third dedup pass:

Image dedup: Perceptual hash (pHash) comparison across imported images.

media.phash is written on upload (routes/media.ts:78-84) and never read. Nothing compares it — no query in the codebase selects or joins on phash. So the pass doesn't exist at all.

2. It isn't a perceptual hash. routes/media.ts:78:

const phash = createHash('sha256').update(fileBuffer).digest('hex');

That's a cryptographic hash of the raw bytes, stored in a column called phash. It only matches byte-identical files — re-encode, resize, or re-compress the same image and it changes completely, which is precisely what a perceptual hash exists to survive. So even if a comparison were added, it would only catch exact file duplicates, not "the same frame grabbed twice".

(CLAUDE.md is honest about this — v2 P9 says "SHA-256 phash on media upload" — but the column name and the spec both say perceptual hash. The name is the misleading part.)

Impact

Low. Nothing breaks; a documented feature simply isn't there, and the column name misleads whoever implements it next into thinking the hard part is done. Given this is a media-research tool where the same music-video frame plausibly gets captured more than once, real image dedup has genuine value — but it's a feature, not a bug fix.

Fix (decide first)

Implement it — swap SHA-256 for an actual perceptual hash (dHash/aHash/pHash; sharp is already a dependency for thumbnails), add a Hamming-distance comparison on upload/import, and route near-matches to the existing duplicate_queue. Needs a migration to recompute or invalidate existing phash values, since the stored ones are not comparable to perceptual hashes.
Or drop it — remove the pass from the spec and rename the column to content_sha256, which is what it is and is independently useful (exact-duplicate detection, integrity checks).

Either way, rename the column — phash holding a SHA-256 is a trap.

Acceptance criteria

The spec and the code agree on whether image dedup exists.
The column name describes what it stores.
(If implementing) a re-encoded copy of an existing image is flagged to duplicate_queue; an unrelated image is not.

Notes

Found while implementing #99 (semantic dedup Pass 2) in v7.2.0 — Pass 2 is now real, which left Pass 3 as the last dedup claim without code. Filed separately: #99 was scoped to text/semantic dedup, and this needs its own product decision plus a migration.

**Severity:** Low · **Confidence:** High · **Effort:** M · Category: code (+ docs drift) ## Problem Same class as #99 — a documented dedup pass that isn't implemented. Two separate issues: **1. There is no image dedup.** The spec (`Tea Leaves - spec.md:240`) lists it as the third dedup pass: > 3. **Image dedup:** Perceptual hash (pHash) comparison across imported images. `media.phash` is **written on upload** (`routes/media.ts:78-84`) and **never read**. Nothing compares it — no query in the codebase selects or joins on `phash`. So the pass doesn't exist at all. **2. It isn't a perceptual hash.** `routes/media.ts:78`: ```ts const phash = createHash('sha256').update(fileBuffer).digest('hex'); ``` That's a cryptographic hash of the raw bytes, stored in a column called `phash`. It only matches **byte-identical** files — re-encode, resize, or re-compress the same image and it changes completely, which is precisely what a perceptual hash exists to survive. So even if a comparison were added, it would only catch exact file duplicates, not "the same frame grabbed twice". (`CLAUDE.md` is honest about this — v2 P9 says "SHA-256 phash on media upload" — but the column name and the spec both say perceptual hash. The name is the misleading part.) ## Impact Low. Nothing breaks; a documented feature simply isn't there, and the column name misleads whoever implements it next into thinking the hard part is done. Given this is a media-research tool where the same music-video frame plausibly gets captured more than once, real image dedup has genuine value — but it's a feature, not a bug fix. ## Fix (decide first) 1. **Implement it** — swap SHA-256 for an actual perceptual hash (dHash/aHash/pHash; `sharp` is already a dependency for thumbnails), add a Hamming-distance comparison on upload/import, and route near-matches to the existing `duplicate_queue`. Needs a migration to recompute or invalidate existing `phash` values, since the stored ones are not comparable to perceptual hashes. 2. **Or drop it** — remove the pass from the spec and rename the column to `content_sha256`, which is what it is and is independently useful (exact-duplicate detection, integrity checks). Either way, **rename the column** — `phash` holding a SHA-256 is a trap. ## Acceptance criteria - [ ] The spec and the code agree on whether image dedup exists. - [ ] The column name describes what it stores. - [ ] (If implementing) a re-encoded copy of an existing image is flagged to `duplicate_queue`; an unrelated image is not. ## Notes Found while implementing #99 (semantic dedup Pass 2) in v7.2.0 — Pass 2 is now real, which left Pass 3 as the last dedup claim without code. Filed separately: #99 was scoped to text/semantic dedup, and this needs its own product decision plus a migration.

claude-bot referenced this issue

2026-07-17 03:09:08 +00:00

Semantic dedup Pass 2 is dead code and imports never embed - implement or correct the docs #99

rbrooks referenced this issue from a commit

2026-07-17 15:54:29 +00:00

feat(v7.2): real perceptual image dedup (#136)

claude-bot commented

2026-07-17 16:01:23 +00:00

Author

Contributor

Implemented in 8bd6f0a (v7.2.0). Real perceptual hashing, and I paid close attention to your "everything matched everything" warning — the threshold is set from measurement, not a guess.

What shipped

Migration 046: phash → content_sha256 (kept — it is a SHA-256, useful for exact-dup + integrity), and a new dhash bit(64).
lib/imageHash.ts: a 64-bit dHash (difference hash) via sharp — resize to 9×8 grayscale, compare each pixel to its right neighbour. dHash encodes the horizontal gradient, so it survives re-encoding/scaling/brightness. I deliberately did not use aHash (average hash), which collides on images of similar overall brightness — that's the classic cause of the failure you hit.
On upload the image is compared against your existing media: exact (content_sha256) then perceptual (bit_count(dhash # $1) Hamming distance). A near-duplicate comes back as duplicate_of: { kind, media_id, entry_id, distance } in the response.
sharp promoted from a transitive dependency to a direct one (it was already pulled in by @huggingface/transformers; relying on that was fragile). Lockfile regenerated.

The threshold — measured, to avoid over-broadness

I instrumented it on real (sharp-generated, smooth) images rather than pick a number:

	Hamming distance (of 64)
Same image, lossy re-encode (PNG↔JPEG q85)	0–7
Genuinely different images	≥ 24 (random ≈ 32)

A 14-bit empty gap. Default media.imageDedupMaxDistance = 10 — inside the gap: catches real re-encodes (≤7) with margin, and sits far below the distinct-image floor (24), so unrelated images are not flagged. (I'd initially set 5; the data showed that misses genuine re-encodes, so 10 it is.) Configurable, capped at 20 to stay under that floor. End-to-end check on the built image: re-encoded copy → distance 0 (flagged), unrelated → 32 (not flagged).

Acceptance criteria

The spec and the code agree on whether image dedup exists — spec + CLAUDE.md updated to describe dHash.
The column name describes what it stores — phash (holding a SHA) → content_sha256.
A re-encoded copy is flagged; an unrelated image is not — covered by imageHash.test.ts (algorithm: re-encode ≤10, distinct >10 & avg well above) and imageDedup.test.ts (the DB/Hamming path: exact, near-dup flagged, beyond-threshold not flagged, setting honoured, closest-wins, self-excluded).

One deviation, flagged rather than guessed

The issue suggested routing near-matches to duplicate_queue. I didn't — that table is structurally import-only (import_job_id NOT NULL, candidate = a post), and image uploads aren't imports; forcing media in would need a schema change and new review-queue UI to render image candidates. Instead the duplicate is returned in the upload response (duplicate_of) — non-destructive, no false schema fit, and the upload still succeeds so nothing is lost. The frontend doesn't yet surface it — a "similar to an existing image" warning on the upload flow is a small follow-up. If you'd rather have a proper media review queue, that's a UI decision worth its own issue; say the word and I'll open it.

Verified on the dev server: migration 046 applies, content_sha256/dhash live on the real DB, health ok, tsc clean, 261/261. All 3 CI jobs green (including the image build with sharp now a direct dep).

Implemented in `8bd6f0a` (v7.2.0). Real perceptual hashing, and I paid close attention to your "everything matched everything" warning — the threshold is set from measurement, not a guess. ### What shipped - **Migration 046**: `phash` → `content_sha256` (kept — it *is* a SHA-256, useful for exact-dup + integrity), and a new `dhash bit(64)`. - **`lib/imageHash.ts`**: a 64-bit **dHash** (difference hash) via `sharp` — resize to 9×8 grayscale, compare each pixel to its right neighbour. dHash encodes the horizontal gradient, so it survives re-encoding/scaling/brightness. I deliberately did **not** use aHash (average hash), which collides on images of similar overall brightness — that's the classic cause of the failure you hit. - **On upload** the image is compared against your existing media: exact (`content_sha256`) then perceptual (`bit_count(dhash # $1)` Hamming distance). A near-duplicate comes back as `duplicate_of: { kind, media_id, entry_id, distance }` in the response. - **`sharp`** promoted from a transitive dependency to a direct one (it was already pulled in by `@huggingface/transformers`; relying on that was fragile). Lockfile regenerated. ### The threshold — measured, to avoid over-broadness I instrumented it on real (sharp-generated, smooth) images rather than pick a number: | | Hamming distance (of 64) | |---|---| | Same image, lossy re-encode (PNG↔JPEG q85) | **0–7** | | Genuinely different images | **≥ 24** (random ≈ 32) | A **14-bit empty gap**. Default `media.imageDedupMaxDistance` = **10** — inside the gap: catches real re-encodes (≤7) with margin, and sits far below the distinct-image floor (24), so unrelated images are **not** flagged. (I'd initially set 5; the data showed that misses genuine re-encodes, so 10 it is.) Configurable, capped at 20 to stay under that floor. End-to-end check on the built image: re-encoded copy → distance **0** (flagged), unrelated → **32** (not flagged). ### Acceptance criteria - [x] The spec and the code agree on whether image dedup exists — spec + CLAUDE.md updated to describe dHash. - [x] The column name describes what it stores — `phash` (holding a SHA) → `content_sha256`. - [x] A re-encoded copy is flagged; an unrelated image is not — covered by `imageHash.test.ts` (algorithm: re-encode ≤10, distinct >10 & avg well above) and `imageDedup.test.ts` (the DB/Hamming path: exact, near-dup flagged, beyond-threshold **not** flagged, setting honoured, closest-wins, self-excluded). ### One deviation, flagged rather than guessed The issue suggested routing near-matches to **`duplicate_queue`**. I didn't — that table is structurally import-only (`import_job_id NOT NULL`, `candidate` = a post), and image uploads aren't imports; forcing media in would need a schema change *and* new review-queue UI to render image candidates. Instead the duplicate is returned in the upload response (`duplicate_of`) — non-destructive, no false schema fit, and the upload still succeeds so nothing is lost. **The frontend doesn't yet surface it** — a "similar to an existing image" warning on the upload flow is a small follow-up. If you'd rather have a proper media review queue, that's a UI decision worth its own issue; say the word and I'll open it. Verified on the dev server: migration 046 applies, `content_sha256`/`dhash` live on the real DB, health `ok`, tsc clean, **261/261**. All 3 CI jobs green (including the image build with `sharp` now a direct dep).

claude-bot closed this issue

2026-07-17 16:01:24 +00:00

rbrooks referenced this issue from a commit

2026-07-17 16:37:40 +00:00

chore(release): v7.2.0 — Reliability, data integrity & docs

rbrooks referenced this issue from a commit

2026-07-17 20:27:24 +00:00

feat(deploy): auto-deploy scaffolding (#134) + image-dedup upload warning (#136)

rbrooks referenced this issue from a commit

2026-07-17 21:38:38 +00:00

chore(release): v7.2.1 — image-dedup upload warning + auto-deploy