Revisit local embedding model (EmbeddingGemma-300M) #27

New issue

Open

opened 2026-06-22 23:54:55 +00:00 by rbrooks · 0 comments

rbrooks commented

2026-06-22 23:54:55 +00:00

Owner

Evaluate when EmbeddingGemma has ~12 months of Node.js / Transformers.js production deployments behind it (roughly mid-to-late 2027). v6 Local Embeddings ships with nomic-embed-text-v1.5 (Apache 2.0, 768-dim, battle-tested).

Why it may be worth switching: MTEB English ~69.67 vs ~62 for nomic-v1.5; same 768-dim (no migration); on-device focus; available via ONNX and Ollama.

Why we waited: Gemma Terms of Use (not Apache 2.0); no fp16 (q8 ~300MB vs ~90MB nomic); community reports of quality regression in some Node deployments as of mid-2026.

Decision criteria: (1) track record in server-side Transformers.js; (2) a q4/q8 QAT variant under ~150MB; (3) no new license restrictions affecting self-hosted use.

Cost if we switch: change default ai.localModel in services/ai.ts + Settings label; existing users re-run bulk re-analysis to regenerate embeddings (surface as a prominent admin action). vector(768) dimension unchanged.

Imported from ROADMAP.md.

Evaluate when EmbeddingGemma has ~12 months of Node.js / Transformers.js production deployments behind it (roughly mid-to-late 2027). v6 Local Embeddings ships with `nomic-embed-text-v1.5` (Apache 2.0, 768-dim, battle-tested). **Why it may be worth switching:** MTEB English ~69.67 vs ~62 for nomic-v1.5; same 768-dim (no migration); on-device focus; available via ONNX and Ollama. **Why we waited:** Gemma Terms of Use (not Apache 2.0); no fp16 (q8 ~300MB vs ~90MB nomic); community reports of quality regression in some Node deployments as of mid-2026. **Decision criteria:** (1) track record in server-side Transformers.js; (2) a q4/q8 QAT variant under ~150MB; (3) no new license restrictions affecting self-hosted use. **Cost if we switch:** change default `ai.localModel` in `services/ai.ts` + Settings label; existing users re-run bulk re-analysis to regenerate embeddings (surface as a prominent admin action). `vector(768)` dimension unchanged. --- _Imported from `ROADMAP.md`._

rbrooks added the

future

label

2026-06-22 23:54:55 +00:00