v9 P2: Real-time ingestion (streaming) #23

New issue

Open

opened 2026-06-22 23:54:53 +00:00 by rbrooks · 0 comments

rbrooks commented

2026-06-22 23:54:53 +00:00

Owner

Replace polling with push wherever the platform actually supports it; keep fast polling where it does not, rather than forcing a worse architecture for an honest gap.

Mastodon (accounts + tags): native streaming API (/api/v1/streaming/..., websocket) replaces polling entirely. A new mastodonStreamManager service holds long-lived connections (one per distinct instance+target where possible), with reconnect/backoff, started alongside the existing workers in index.ts.
Bluesky accounts: a single shared Jetstream consumer per Tea Leaves instance (not per watch), wantedCollections=app.bsky.feed.post, wantedDids=<every watched account DID across all projects>, updated dynamically as watches change. Jetstream filters server-side by account, so this is genuinely real-time and cheap regardless of watch count.
Bluesky tags/keywords: stay on polling, just much faster (watch.bskyTagPollSeconds, default 30-60s). Jetstream has no server-side text filter; true real-time keyword tracking would mean consuming the full unfiltered firehose and matching client-side - disproportionate versus a 30-60s poll of the search API.
Operational surface: these are long-lived connections, a different runtime shape than the short, stateless BullMQ jobs used everywhere else. They need explicit lifecycle management (start on boot, reconnect with backoff, clean shutdown on SIGTERM).
Surface connection health in the UI (a small "connected / reconnecting" indicator near the Watch nav item) so a silently-dropped firehose does not masquerade as "no new posts."

Originated in ROADMAP.md (now retired; this tracker is the single source of truth).

Replace polling with push wherever the platform actually supports it; keep fast polling where it does not, rather than forcing a worse architecture for an honest gap. - [ ] **Mastodon (accounts + tags):** native streaming API (`/api/v1/streaming/...`, websocket) replaces polling entirely. A new `mastodonStreamManager` service holds long-lived connections (one per distinct instance+target where possible), with reconnect/backoff, started alongside the existing workers in `index.ts`. - [ ] **Bluesky accounts:** a single shared **Jetstream** consumer per Tea Leaves instance (not per watch), `wantedCollections=app.bsky.feed.post`, `wantedDids=<every watched account DID across all projects>`, updated dynamically as watches change. Jetstream filters server-side by account, so this is genuinely real-time and cheap regardless of watch count. - [ ] **Bluesky tags/keywords:** stay on polling, just much faster (`watch.bskyTagPollSeconds`, default 30-60s). Jetstream has no server-side text filter; true real-time keyword tracking would mean consuming the full unfiltered firehose and matching client-side - disproportionate versus a 30-60s poll of the search API. - [ ] **Operational surface:** these are long-lived connections, a different runtime shape than the short, stateless BullMQ jobs used everywhere else. They need explicit lifecycle management (start on boot, reconnect with backoff, clean shutdown on `SIGTERM`). - [ ] Surface connection health in the UI (a small "connected / reconnecting" indicator near the Watch nav item) so a silently-dropped firehose does not masquerade as "no new posts." --- _Originated in `ROADMAP.md` (now retired; this tracker is the single source of truth)._

rbrooks added this to the v9.0.0 - Social Monitoring & Quick Capture milestone

2026-06-22 23:54:53 +00:00

rbrooks added the

enhancement

label

2026-06-22 23:54:53 +00:00