rbrooks/TeaLeaves

Watch

Fork 0

Semantic dedup Pass 2 is dead code and imports never embed - implement or correct the docs #99

New issue

Closed

opened 2026-07-15 19:51:32 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-15 19:51:32 +00:00

Contributor

Severity: Medium · Confidence: High · Effort: M · Category: code (+ docs drift)

Evidence

api/src/services/dedup.ts:17-33 - only exact source_id match; the flag action is declared in DedupResult (:15) but never returned; comment at :19 says "Pass 2 . deferred to Phase 4".
api/src/services/importWorker.ts:262-272 - the flag → duplicate_queue branch (dead); :278 - INSERT INTO entries with no embedEntry call.
CLAUDE.md claims "Pass 2 (active): cosine similarity on text_embedding . → flag to duplicate_queue"; spec docs/Tea Leaves - spec.md:237-242.

Problem
Two documented behaviors don't exist. (1) Semantic dedup Pass 2 is not implemented - checkDedup never returns flag, so duplicate_queue is never populated by similarity; the import worker's flag branch is unreachable. (2) Imported entries are inserted without embeddings, so even a future Pass 2 would see NULL vectors until a manual reanalyze.

Impact
A documented core feature silently no-ops (the duplicate review queue never fills from similarity), and semantic search / "more like this" over imported content is empty until the user runs a bulk reanalyze. The bigger cost is trust: the docs assert behavior the code lacks.

Fix (two parts)

Short term - remove the dead flag branch/type arm and correct CLAUDE.md + spec to state Pass 2 is deferred.
Tracked here for implementation - embed on import (or enqueue embedding after insert), run cosine over text_embedding above search.similarityThreshold, populate duplicate_queue for review.

Acceptance criteria

Docs and code agree on whether Pass 2 exists.
(If implementing) imported entries get embeddings and near-duplicates land in the review queue.

Filed from the 2026-07-15 codebase audit. Full report: docs/.internal/report-2026-07-15.md (gitignored).

**Severity:** Medium · **Confidence:** High · **Effort:** M · Category: code (+ docs drift) **Evidence** - `api/src/services/dedup.ts:17-33` - only exact `source_id` match; the `flag` action is declared in `DedupResult` (:15) but never returned; comment at :19 says "Pass 2 . deferred to Phase 4". - `api/src/services/importWorker.ts:262-272` - the `flag` → `duplicate_queue` branch (dead); `:278` - `INSERT INTO entries` with no `embedEntry` call. - `CLAUDE.md` claims "Pass 2 (active): cosine similarity on `text_embedding` . → flag to `duplicate_queue`"; spec `docs/Tea Leaves - spec.md:237-242`. **Problem** Two documented behaviors don't exist. (1) Semantic dedup Pass 2 is not implemented - `checkDedup` never returns `flag`, so `duplicate_queue` is never populated by similarity; the import worker's flag branch is unreachable. (2) Imported entries are inserted without embeddings, so even a future Pass 2 would see NULL vectors until a manual reanalyze. **Impact** A documented core feature silently no-ops (the duplicate review queue never fills from similarity), and semantic search / "more like this" over imported content is empty until the user runs a bulk reanalyze. The bigger cost is trust: the docs assert behavior the code lacks. **Fix (two parts)** 1. Short term - remove the dead flag branch/type arm and correct CLAUDE.md + spec to state Pass 2 is deferred. 2. Tracked here for implementation - embed on import (or enqueue embedding after insert), run cosine over `text_embedding` above `search.similarityThreshold`, populate `duplicate_queue` for review. **Acceptance criteria** - [ ] Docs and code agree on whether Pass 2 exists. - [ ] (If implementing) imported entries get embeddings and near-duplicates land in the review queue. --- _Filed from the 2026-07-15 codebase audit. Full report: `docs/.internal/report-2026-07-15.md` (gitignored)._

claude-bot added this to the v7.2.0 - Reliability, data integrity & docs milestone

2026-07-15 19:51:32 +00:00

claude-bot added the

bug

label

2026-07-15 19:51:32 +00:00

claude-bot referenced this issue

2026-07-16 22:05:06 +00:00

Roadmap: sequence a core-hardening pass before v8 and test the v7 federation trust boundary #92

rbrooks referenced this issue from a commit

2026-07-17 02:59:46 +00:00

feat(v7.2): implement semantic dedup Pass 2 + embed on import (#99)

claude-bot referenced this issue

2026-07-17 03:00:03 +00:00

Image dedup (spec Pass 3) doesn't exist, and media.phash is a SHA-256 of the bytes, not a perceptual hash #136

claude-bot commented

2026-07-17 03:09:08 +00:00

Author

Contributor

Implemented in 26ba1f0 (v7.2.0). Went with implement, not defer — the duplicate_queue review UI has existed since Phase 3 and could never fill, and the schema already had similarity FLOAT and dedup_type CHECK IN ('source_id', 'semantic') waiting for it.

Pass 2 is real. checkDedup runs cosine similarity over text_embedding and returns { action: 'flag', reason: 'semantic_similarity', existingId, similarity }; the import worker's previously-unreachable branch now writes duplicate_queue with dedup_type='semantic' and the score. Existing rows with no vector, and soft-deleted ones, are excluded from the comparison.

Imports embed inline. The candidate's vector is computed once in checkDedup (Pass 2 needs it) and returned on the import path, so the worker stores it on the INSERT — one AI call per post serving both the check and the stored embedding. Semantic search / "more like this" now works over imported archives immediately, no bulk reanalyze needed.

AI stays optional: no embedding provider → generateTextEmbedding returns null → Pass 2 is skipped silently and the import behaves exactly as before.

One deliberate deviation from the issue

The issue says compare "above search.similarityThreshold". I didn't — that default is 0.65, which answers "what is related?". Dedup needs "is this the same?". At 0.65, importing a themed archive would flag a large share of its own posts as duplicates and drown the review queue, which is worse than having no Pass 2. Added a dedicated import.dedupThreshold (default 0.95), live-editable and validated like the other settings.

A test pins the distinction: an entry at 0.707 similarity must not flag — and 0.707 is above 0.65, so it is precisely the false positive the search threshold would have produced. If you'd rather have the looser behaviour it's now one setting change, not a code change.

Also removed

backfillEmbeddings() — zero callers, despite its own doc comment claiming it was "Called at import completion". It was evidently the intended embed-on-import mechanism and was never wired up. It's redundant with runEmbeddingsJob (POST /api/ai/reanalyze { type: 'embeddings' }), which does the same work with progress tracking and suggestion enqueueing.

Acceptance criteria:

Docs and code agree on whether Pass 2 exists — CLAUDE.md now describes the real behaviour, thresholds and the AI-optional path. The spec (Tea Leaves - spec.md:238-239) needed no change: it described Passes 1–2 accurately and is now true rather than aspirational.
Imported entries get embeddings and near-duplicates land in the review queue.

Tests — api/src/test/integration/dedupPass2.test.ts, 7 cases with generateTextEmbedding mocked so Pass 2 is deterministic without an embedding server: Pass 1 short-circuits (and never calls the embedder), identical → flag with similarity ≈1, orthogonal → import returning the vector, no-vector/soft-deleted rows ignored, 0.707 → no flag at 0.95, 0.707 → flag at 0.5 (threshold honoured), AI unavailable → import.

Uses one-hot unit vectors so similarities are exact (1.0 / 0.0 / 0.7071) rather than approximate.

CI run 194 green — 249/249 tests. Deployed to dev.

Follow-up filed: #136. With Pass 2 real, the spec's Pass 3 (image dedup) is now the last dedup claim without code — media.phash is written and never read, and it's a SHA-256 of the file bytes, not a perceptual hash. Left unmilestoned pending your decision (implement vs. drop + rename the column).

Implemented in `26ba1f0` (v7.2.0). Went with **implement**, not defer — the `duplicate_queue` review UI has existed since Phase 3 and could never fill, and the schema already had `similarity FLOAT` and `dedup_type CHECK IN ('source_id', 'semantic')` waiting for it. **Pass 2 is real.** `checkDedup` runs cosine similarity over `text_embedding` and returns `{ action: 'flag', reason: 'semantic_similarity', existingId, similarity }`; the import worker's previously-unreachable branch now writes `duplicate_queue` with `dedup_type='semantic'` and the score. Existing rows with no vector, and soft-deleted ones, are excluded from the comparison. **Imports embed inline.** The candidate's vector is computed **once** in `checkDedup` (Pass 2 needs it) and returned on the import path, so the worker stores it on the `INSERT` — one AI call per post serving both the check and the stored embedding. Semantic search / "more like this" now works over imported archives immediately, no bulk reanalyze needed. **AI stays optional:** no embedding provider → `generateTextEmbedding` returns null → Pass 2 is skipped silently and the import behaves exactly as before. ### One deliberate deviation from the issue The issue says compare "above `search.similarityThreshold`". **I didn't** — that default is **0.65**, which answers *"what is related?"*. Dedup needs *"is this the same?"*. At 0.65, importing a themed archive would flag a large share of its own posts as duplicates and drown the review queue, which is worse than having no Pass 2. Added a dedicated **`import.dedupThreshold` (default 0.95)**, live-editable and validated like the other settings. A test pins the distinction: an entry at **0.707** similarity must **not** flag — and 0.707 is above 0.65, so it is precisely the false positive the search threshold would have produced. If you'd rather have the looser behaviour it's now one setting change, not a code change. ### Also removed `backfillEmbeddings()` — **zero callers**, despite its own doc comment claiming it was "Called at import completion". It was evidently the intended embed-on-import mechanism and was never wired up. It's redundant with `runEmbeddingsJob` (`POST /api/ai/reanalyze { type: 'embeddings' }`), which does the same work with progress tracking and suggestion enqueueing. **Acceptance criteria:** - [x] Docs and code agree on whether Pass 2 exists — `CLAUDE.md` now describes the real behaviour, thresholds and the AI-optional path. The spec (`Tea Leaves - spec.md:238-239`) needed no change: it described Passes 1–2 accurately and is now true rather than aspirational. - [x] Imported entries get embeddings and near-duplicates land in the review queue. **Tests** — `api/src/test/integration/dedupPass2.test.ts`, 7 cases with `generateTextEmbedding` mocked so Pass 2 is deterministic without an embedding server: Pass 1 short-circuits (and never calls the embedder), identical → flag with similarity ≈1, orthogonal → import returning the vector, no-vector/soft-deleted rows ignored, 0.707 → no flag at 0.95, 0.707 → flag at 0.5 (threshold honoured), AI unavailable → import. Uses one-hot unit vectors so similarities are exact (1.0 / 0.0 / 0.7071) rather than approximate. CI run 194 green — 249/249 tests. Deployed to dev. --- **Follow-up filed: #136.** With Pass 2 real, the spec's **Pass 3** (image dedup) is now the last dedup claim without code — `media.phash` is written and never read, and it's a **SHA-256 of the file bytes**, not a perceptual hash. Left unmilestoned pending your decision (implement vs. drop + rename the column).

claude-bot closed this issue

2026-07-17 03:09:24 +00:00