v9 P2: Real-time ingestion (streaming) #23

New issue

Open

opened 2026-06-22 23:54:53 +00:00 by rbrooks · 1 comment

rbrooks commented

2026-06-22 23:54:53 +00:00

Owner

Replace polling with push wherever the platform actually supports it; keep fast polling where it does not, rather than forcing a worse architecture for an honest gap.

Mastodon (accounts + tags): native streaming API (/api/v1/streaming/..., websocket) replaces polling entirely. A new mastodonStreamManager service holds long-lived connections (one per distinct instance+target where possible), with reconnect/backoff, started alongside the existing workers in index.ts.
Bluesky accounts: a single shared Jetstream consumer per Tea Leaves instance (not per watch), wantedCollections=app.bsky.feed.post, wantedDids=<every watched account DID across all projects>, updated dynamically as watches change. Jetstream filters server-side by account, so this is genuinely real-time and cheap regardless of watch count.
Bluesky tags/keywords: stay on polling, just much faster (watch.bskyTagPollSeconds, default 30-60s). Jetstream has no server-side text filter; true real-time keyword tracking would mean consuming the full unfiltered firehose and matching client-side - disproportionate versus a 30-60s poll of the search API.
Operational surface: these are long-lived connections, a different runtime shape than the short, stateless BullMQ jobs used everywhere else. They need explicit lifecycle management (start on boot, reconnect with backoff, clean shutdown on SIGTERM).
Surface connection health in the UI (a small "connected / reconnecting" indicator near the Watch nav item) so a silently-dropped firehose does not masquerade as "no new posts."

Originated in ROADMAP.md (now retired; this tracker is the single source of truth).

Replace polling with push wherever the platform actually supports it; keep fast polling where it does not, rather than forcing a worse architecture for an honest gap. - [ ] **Mastodon (accounts + tags):** native streaming API (`/api/v1/streaming/...`, websocket) replaces polling entirely. A new `mastodonStreamManager` service holds long-lived connections (one per distinct instance+target where possible), with reconnect/backoff, started alongside the existing workers in `index.ts`. - [ ] **Bluesky accounts:** a single shared **Jetstream** consumer per Tea Leaves instance (not per watch), `wantedCollections=app.bsky.feed.post`, `wantedDids=<every watched account DID across all projects>`, updated dynamically as watches change. Jetstream filters server-side by account, so this is genuinely real-time and cheap regardless of watch count. - [ ] **Bluesky tags/keywords:** stay on polling, just much faster (`watch.bskyTagPollSeconds`, default 30-60s). Jetstream has no server-side text filter; true real-time keyword tracking would mean consuming the full unfiltered firehose and matching client-side - disproportionate versus a 30-60s poll of the search API. - [ ] **Operational surface:** these are long-lived connections, a different runtime shape than the short, stateless BullMQ jobs used everywhere else. They need explicit lifecycle management (start on boot, reconnect with backoff, clean shutdown on `SIGTERM`). - [ ] Surface connection health in the UI (a small "connected / reconnecting" indicator near the Watch nav item) so a silently-dropped firehose does not masquerade as "no new posts." --- _Originated in `ROADMAP.md` (now retired; this tracker is the single source of truth)._

rbrooks added this to the v9.0.0 - Social Monitoring & Quick Capture milestone

2026-06-22 23:54:53 +00:00

rbrooks added the

enhancement

label

2026-06-22 23:54:53 +00:00

claude-bot referenced this issue

2026-07-18 06:18:03 +00:00

v9.0.0 — Social Monitoring & Quick Capture #139

rbrooks referenced this issue from a commit

2026-07-18 06:30:45 +00:00

feat(v9 p2): Bluesky Jetstream real-time consumer (#23)

claude-bot commented

2026-07-18 06:44:24 +00:00

Contributor

Partially shipped in v9.0.0 (PR #139):

Bluesky accounts → Jetstream: done. A single shared Jetstream WS per instance, server-side wantedDids filter over every watched account DID, reconnect/backoff, cursor resume, dynamic DID set. Gated by watch.streamingEnabled (default off); additive to polling (global source_id dedup). Health at GET /api/watch/stream-status. Pure event parser unit-tested; the live socket needs deploy verification.

Remaining (deferred to the P3 spike):

Mastodon streaming: most instances no longer allow unauthenticated streaming, so real-time Mastodon needs the OAuth token from #24 (account linking). Until then Mastodon stays on polling. This is an honest platform-reality deviation from the original plan — real-time Mastodon rides with P3.
Bluesky tags → faster polling (watch.bskyTagPollSeconds): setting is added/validated; wiring a separate faster poll schedule for tag watches is a small follow-up (currently on the main poll interval).
UI connection-health chip: the endpoint exists; the Watch-page indicator is a minor follow-up.

Leaving this open for the Mastodon-streaming half + the two small follow-ups.

**Partially shipped in v9.0.0** (PR #139): - **Bluesky accounts → Jetstream**: done. A single shared Jetstream WS per instance, server-side `wantedDids` filter over every watched account DID, reconnect/backoff, cursor resume, dynamic DID set. Gated by `watch.streamingEnabled` (default off); additive to polling (global `source_id` dedup). Health at `GET /api/watch/stream-status`. Pure event parser unit-tested; the live socket needs deploy verification. **Remaining (deferred to the P3 spike):** - **Mastodon streaming**: most instances no longer allow *unauthenticated* streaming, so real-time Mastodon needs the OAuth token from #24 (account linking). Until then Mastodon stays on polling. This is an honest platform-reality deviation from the original plan — real-time Mastodon rides with P3. - **Bluesky tags → faster polling** (`watch.bskyTagPollSeconds`): setting is added/validated; wiring a separate faster poll schedule for tag watches is a small follow-up (currently on the main poll interval). - **UI connection-health chip**: the endpoint exists; the Watch-page indicator is a minor follow-up. Leaving this open for the Mastodon-streaming half + the two small follow-ups.

claude-bot referenced this issue

2026-07-18 06:44:31 +00:00

v9 P3: Account linking (OAuth) #24