Extract BullMQ workers to a separate process before v9 streaming #117

New issue

Closed

opened 2026-07-15 19:53:41 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-07-15 19:53:41 +00:00

Contributor

Severity: Low · Confidence: High · Category: architecture (v9 prerequisite)

Evidence

api/src/index.ts:17-29 - all BullMQ workers + startBackupScheduler() + app.listen run in one process.
Heavy jobs: ffmpeg/whisper (mediaProcessorWorker), ONNX embeddings (ai.ts local), restic/pg_dump (backup).

Assessment
One process for API + all workers is appropriate for a single-user homelab today and keeps deployment simple. But heavy jobs compete with request handling for the event loop, and a wedged worker (see F-09) can degrade API latency. The v9 plan (Mastodon streaming + Bluesky Jetstream) introduces long-lived connections - a different runtime shape the v9 issue itself flags - that will strain the single-process model more than short BullMQ jobs do.

Recommendation
Keep as-is for now. Before v9 (streaming) starts, extract workers into a second container sharing the codebase (compose service worker) with shared Redis/DB - the natural home for v9's long-lived streams. Honest case for doing nothing: at one user and current volumes, the single process is fine and simpler to operate - so this is a v9 prerequisite, not a standalone task.

Acceptance criteria

When v9 begins, streaming consumers run in a process/container separate from the HTTP API.

Filed from the 2026-07-15 codebase audit. Full report: docs/.internal/report-2026-07-15.md (gitignored).

**Severity:** Low · **Confidence:** High · Category: architecture (v9 prerequisite) **Evidence** - `api/src/index.ts:17-29` - all BullMQ workers + `startBackupScheduler()` + `app.listen` run in one process. - Heavy jobs: ffmpeg/whisper (`mediaProcessorWorker`), ONNX embeddings (`ai.ts` local), restic/pg_dump (`backup`). **Assessment** One process for API + all workers is appropriate for a single-user homelab today and keeps deployment simple. But heavy jobs compete with request handling for the event loop, and a wedged worker (see F-09) can degrade API latency. The v9 plan (Mastodon streaming + Bluesky Jetstream) introduces long-lived connections - a different runtime shape the v9 issue itself flags - that will strain the single-process model more than short BullMQ jobs do. **Recommendation** Keep as-is for now. Before v9 (streaming) starts, extract workers into a second container sharing the codebase (compose service `worker`) with shared Redis/DB - the natural home for v9's long-lived streams. Honest case for doing nothing: at one user and current volumes, the single process is fine and simpler to operate - so this is a v9 prerequisite, not a standalone task. **Acceptance criteria** - [ ] When v9 begins, streaming consumers run in a process/container separate from the HTTP API. --- _Filed from the 2026-07-15 codebase audit. Full report: `docs/.internal/report-2026-07-15.md` (gitignored)._

claude-bot added this to the v9.0.0 - Social Monitoring & Quick Capture milestone

2026-07-15 19:53:41 +00:00

claude-bot added the

enhancement

label

2026-07-15 19:53:41 +00:00

claude-bot referenced this issue

2026-07-15 20:35:40 +00:00

[Scope] Real-time multi-researcher co-editing #133

claude-bot referenced this issue

2026-07-18 05:55:48 +00:00

v9 P1: Public-data watch + one-click capture (polling MVP) #22