OCR / text extraction from screenshots into searchable content #118

New issue

Closed

opened 2026-07-15 19:53:42 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-15 19:53:42 +00:00

Contributor

Feature idea (audit Phase 5, rank #1 by value-to-effort) · Effort: low-medium

Why
The domain is media analysis and the user screenshots lyrics, captions, and on-screen text. Today the vision model produces a description and tags (api/src/services/ai.ts) but never extracts the literal text into the entry's searchable content, so screenshotted text isn't findable by full-text search.

Proposal
Reuse the existing vision provider to OCR image text on upload and offer the extracted text for the user to accept into content - the same approve-before-apply pattern already used for AI tag suggestions (nothing applied automatically). Fits the provider-agnostic AI service and the media pipeline.

Why it was likely missed
AI was scoped to "describe + tag," not "transcribe image text."

Acceptance criteria

On image upload, extracted text is offered for the user to accept into the entry content.
Accepted text is sanitized on write and becomes full-text searchable.
Nothing is applied without explicit user approval.

Filed from the 2026-07-15 codebase audit. Full report: docs/.internal/report-2026-07-15.md (gitignored).

**Feature idea** (audit Phase 5, rank #1 by value-to-effort) · Effort: low-medium **Why** The domain is media analysis and the user screenshots lyrics, captions, and on-screen text. Today the vision model produces a *description* and *tags* (`api/src/services/ai.ts`) but never extracts the literal text into the entry's searchable content, so screenshotted text isn't findable by full-text search. **Proposal** Reuse the existing vision provider to OCR image text on upload and offer the extracted text for the user to accept into `content` - the same approve-before-apply pattern already used for AI tag suggestions (nothing applied automatically). Fits the provider-agnostic AI service and the media pipeline. **Why it was likely missed** AI was scoped to "describe + tag," not "transcribe image text." **Acceptance criteria** - [ ] On image upload, extracted text is offered for the user to accept into the entry content. - [ ] Accepted text is sanitized on write and becomes full-text searchable. - [ ] Nothing is applied without explicit user approval. --- _Filed from the 2026-07-15 codebase audit. Full report: `docs/.internal/report-2026-07-15.md` (gitignored)._

claude-bot added this to the v10.0.0 - Research workflow enhancements milestone

2026-07-15 19:53:42 +00:00

claude-bot added the

enhancement

label

2026-07-15 19:53:42 +00:00

claude-bot referenced this issue from a pull request that will close it,

2026-07-18 16:57:12 +00:00

v9.3.0 — OCR (#118) + import adapters (#121) + offline read cache (#122) #142

claude-bot closed this issue

2026-07-18 17:11:29 +00:00

claude-bot referenced this issue from a commit

2026-07-18 17:11:30 +00:00

Merge pull request 'v9.3.0 — OCR (#118) + import adapters (#121) + offline read cache (#122)' (#142) from feat/v10-utilities into main

claude-bot commented

2026-07-18 17:12:04 +00:00

Author

Contributor

Done — shipped in v9.3.0 (PR #142, merge c69897c).

extractImageText() reuses the provider-agnostic vision path (new OCR prompt); runAIAnalysis stores media.ocr_text (migration 055). POST /api/media/:id/accept-ocr appends the text to the entry content — sanitized on write, and full-text searchable immediately (the FTS index is an expression over entries.content). UI: a "Show extracted text (OCR)" → "Add transcribed text to entry" control on the media card. 3 integration tests.

Acceptance: ✅ extracted text offered on upload · ✅ accepted text sanitized + searchable · ✅ nothing applied without explicit approval (no-ops entirely when AI isn't configured).

Done — shipped in **v9.3.0** (PR #142, merge `c69897c`). `extractImageText()` reuses the provider-agnostic vision path (new OCR prompt); `runAIAnalysis` stores `media.ocr_text` (migration 055). `POST /api/media/:id/accept-ocr` appends the text to the entry content — **sanitized on write**, and full-text searchable immediately (the FTS index is an expression over `entries.content`). UI: a "Show extracted text (OCR)" → "Add transcribed text to entry" control on the media card. 3 integration tests. **Acceptance:** ✅ extracted text offered on upload · ✅ accepted text sanitized + searchable · ✅ nothing applied without explicit approval (no-ops entirely when AI isn't configured).