rbrooks/TeaLeaves
Watch
1
0
Fork
You've already forked TeaLeaves
 0
Code Issues 14 Pull requests 2 Projects Releases 17 Packages Wiki Activity Actions

v9 P1: Public-data watch + one-click capture (polling MVP) #22

New issue
Closed
opened 2026-06-22 23:54:53 +00:00 by rbrooks · 1 comment
rbrooks commented 2026-06-22 23:54:53 +00:00
Owner
Copy link

Baseline Watch feature: no account linking, public data only, works immediately. P1 alone is a complete, useful feature.

Platform reality check (sets expectations, not a blocker):

  • Bluesky has a public AppView API and effectively one global index, so both account and keyword/hashtag watching have real reach (app.bsky.feed.getAuthorFeed, app.bsky.feed.searchPosts).
  • Mastodon / the fediverse is decentralized - there is no global hashtag index. Hashtag watching only sees what the queried instance has already federated in, so it is inherently a partial view, not a bug. Account watching is reliable (query the account's home instance directly). The UI must say this explicitly next to Mastodon tag watches.

Schema:

  • watches table: id uuid PK, project_id uuid NOT NULL REFERENCES projects ON DELETE CASCADE, platform text NOT NULL CHECK (platform IN ('bluesky','mastodon')), target_type text NOT NULL CHECK (target_type IN ('account','tag')), target_value text NOT NULL (handle/hashtag without @/#), instance_url text (Mastodon only; validated via isSafeUrl()), enabled boolean NOT NULL DEFAULT true, cursor text (pagination/since-token; reused as the P2 stream resume-point), last_seen_at timestamptz, last_error text, created_at timestamptz NOT NULL DEFAULT now(). UNIQUE (project_id, platform, target_type, target_value, instance_url).
  • discovered_posts table: id uuid PK, watch_id uuid NOT NULL REFERENCES watches ON DELETE CASCADE, source_id text NOT NULL (bluesky:<at-uri> / mastodon:<instance_url>:<status_id> - same dedup shape as import source_id), author_handle text, author_display_name text, content text NOT NULL (sanitized on write), post_url text NOT NULL, media_urls jsonb, posted_at timestamptz NOT NULL, status text NOT NULL DEFAULT 'new' CHECK (status IN ('new','added','dismissed')), created_entry_id uuid REFERENCES entries ON DELETE SET NULL, created_at timestamptz NOT NULL DEFAULT now(). UNIQUE (source_id) - doubles as the ingestion dedup guard across all three phases.

Polling worker (services/watchWorker.ts):

  • BullMQ repeatable job, same pattern as federationSyncWorker; interval in Settings (watch.pollIntervalMinutes, default 10)
  • Platform adapters behind a small interface (lib/watchAdapters/bluesky.ts, mastodon.ts), mirroring how importWorker separates parsers
  • Mastodon adapter: resolve target_value via /api/v1/accounts/lookup, then poll /api/v1/accounts/:id/statuses?since_id=; tag watches poll /api/v1/timelines/tag/:hashtag?since_id=
  • Bluesky adapter: public unauthenticated AppView host (public.api.bsky.app) for both account feeds and search

One-click "Add as entry":

  • Shared addSocialPostAsEntry(post, projectId, userId) helper (used by both the queue and direct search) calls the existing checkDedup() BEFORE insert
  • On a dedup hit, surface "already in this project, view existing entry" rather than silently creating a duplicate
  • POST /api/discovered-posts/:id/add (sets status='added', created_entry_id) and POST /api/discovered-posts/:id/dismiss
  • Bulk add-all / dismiss-all deferred unless the per-post queue proves too slow in practice

Direct search (3rd tab on the Watch page, alongside Watches + Queue):

  • GET /api/watch/search?platform=&type=account|tag&q=&instanceUrl= - Bluesky via app.bsky.actor.searchActors / app.bsky.feed.searchPosts; Mastodon via that instance's /api/v2/search + /api/v1/timelines/tag/:hashtag preview. Results rendered live, not persisted.
  • Account/tag result -> "Add to watchlist" pre-fills the watch-creation form (only asks for the project)
  • Individual post -> "Add as entry" calls addSocialPostAsEntry() directly (no discovered_posts row), same dedup-hit handling
  • Mastodon caveat carries over: unauthenticated resolve=true cross-instance lookup typically needs a session, so account search only reliably finds accounts the queried instance already knows. One-line UI note; improves once P3 ships.

Nav + security:

  • New top-level Watch nav item with an unread / pending-review count badge
  • instance_url (user-supplied, watches + direct search) validated via isSafeUrl() with a private-IP-range / SSRF check before the first outbound request to a new instance. All P1 requests are unauthenticated GETs to public endpoints - no credentials in play yet.

Originated in ROADMAP.md (now retired; this tracker is the single source of truth).

Baseline Watch feature: no account linking, public data only, works immediately. P1 alone is a complete, useful feature. **Platform reality check (sets expectations, not a blocker):** - Bluesky has a public AppView API and effectively one global index, so both account *and* keyword/hashtag watching have real reach (`app.bsky.feed.getAuthorFeed`, `app.bsky.feed.searchPosts`). - Mastodon / the fediverse is decentralized - there is no global hashtag index. Hashtag watching only sees what the queried instance has already federated in, so it is inherently a partial view, not a bug. Account watching is reliable (query the account's home instance directly). **The UI must say this explicitly next to Mastodon tag watches.** **Schema:** - [ ] `watches` table: `id uuid PK`, `project_id uuid NOT NULL REFERENCES projects ON DELETE CASCADE`, `platform text NOT NULL CHECK (platform IN ('bluesky','mastodon'))`, `target_type text NOT NULL CHECK (target_type IN ('account','tag'))`, `target_value text NOT NULL` (handle/hashtag without `@`/`#`), `instance_url text` (Mastodon only; validated via `isSafeUrl()`), `enabled boolean NOT NULL DEFAULT true`, `cursor text` (pagination/since-token; reused as the P2 stream resume-point), `last_seen_at timestamptz`, `last_error text`, `created_at timestamptz NOT NULL DEFAULT now()`. `UNIQUE (project_id, platform, target_type, target_value, instance_url)`. - [ ] `discovered_posts` table: `id uuid PK`, `watch_id uuid NOT NULL REFERENCES watches ON DELETE CASCADE`, `source_id text NOT NULL` (`bluesky:<at-uri>` / `mastodon:<instance_url>:<status_id>` - same dedup shape as import `source_id`), `author_handle text`, `author_display_name text`, `content text NOT NULL` (sanitized on write), `post_url text NOT NULL`, `media_urls jsonb`, `posted_at timestamptz NOT NULL`, `status text NOT NULL DEFAULT 'new' CHECK (status IN ('new','added','dismissed'))`, `created_entry_id uuid REFERENCES entries ON DELETE SET NULL`, `created_at timestamptz NOT NULL DEFAULT now()`. `UNIQUE (source_id)` - doubles as the ingestion dedup guard across all three phases. **Polling worker (`services/watchWorker.ts`):** - [ ] BullMQ repeatable job, same pattern as `federationSyncWorker`; interval in Settings (`watch.pollIntervalMinutes`, default 10) - [ ] Platform adapters behind a small interface (`lib/watchAdapters/bluesky.ts`, `mastodon.ts`), mirroring how `importWorker` separates parsers - [ ] Mastodon adapter: resolve `target_value` via `/api/v1/accounts/lookup`, then poll `/api/v1/accounts/:id/statuses?since_id=`; tag watches poll `/api/v1/timelines/tag/:hashtag?since_id=` - [ ] Bluesky adapter: public unauthenticated AppView host (`public.api.bsky.app`) for both account feeds and search **One-click "Add as entry":** - [ ] Shared `addSocialPostAsEntry(post, projectId, userId)` helper (used by both the queue and direct search) calls the existing `checkDedup()` BEFORE insert - [ ] On a dedup hit, surface "already in this project, view existing entry" rather than silently creating a duplicate - [ ] `POST /api/discovered-posts/:id/add` (sets `status='added'`, `created_entry_id`) and `POST /api/discovered-posts/:id/dismiss` - [ ] Bulk add-all / dismiss-all deferred unless the per-post queue proves too slow in practice **Direct search (3rd tab on the Watch page, alongside Watches + Queue):** - [ ] `GET /api/watch/search?platform=&type=account|tag&q=&instanceUrl=` - Bluesky via `app.bsky.actor.searchActors` / `app.bsky.feed.searchPosts`; Mastodon via that instance's `/api/v2/search` + `/api/v1/timelines/tag/:hashtag` preview. Results rendered live, not persisted. - [ ] Account/tag result -> "Add to watchlist" pre-fills the watch-creation form (only asks for the project) - [ ] Individual post -> "Add as entry" calls `addSocialPostAsEntry()` directly (no `discovered_posts` row), same dedup-hit handling - [ ] Mastodon caveat carries over: unauthenticated `resolve=true` cross-instance lookup typically needs a session, so account search only reliably finds accounts the queried instance already knows. One-line UI note; improves once P3 ships. **Nav + security:** - [ ] New top-level **Watch** nav item with an unread / pending-review count badge - [ ] `instance_url` (user-supplied, watches + direct search) validated via `isSafeUrl()` with a private-IP-range / SSRF check before the first outbound request to a new instance. All P1 requests are unauthenticated GETs to public endpoints - no credentials in play yet. --- _Originated in `ROADMAP.md` (now retired; this tracker is the single source of truth)._
claude-bot commented 2026-07-18 05:55:48 +00:00
Contributor
Copy link

Picking this up on a feat/v9-social-watch branch (v8.0.0 is released & deployed).

Build order:

  1. Migration 052: watches + discovered_posts tables (per the schema above).
  2. Platform adapters (lib/watchAdapters/{bluesky,mastodon}.ts) behind a small interface — Bluesky via the public AppView (public.api.bsky.app), Mastodon via instance REST. instance_url gated by isSafeUrl() + the existing SSRF/private-IP guard before any outbound request.
  3. services/watchWorker.ts — BullMQ repeatable job (mirrors federationSyncWorker), interval watch.pollIntervalMinutes (default 10).
  4. addSocialPostAsEntry() shared helper calling checkDedup() before insert; watches CRUD, discovered-posts/:id/add|dismiss, and the live direct-search endpoint.
  5. Frontend: Watch nav item + page (Watches / Queue / Search tabs).

Note: #117 (workers → separate process) is a P2 prerequisite, not P1 — the single-process model is fine for polling, so P1 proceeds as-is.

Picking this up on a `feat/v9-social-watch` branch (v8.0.0 is released & deployed). Build order: 1. Migration 052: `watches` + `discovered_posts` tables (per the schema above). 2. Platform adapters (`lib/watchAdapters/{bluesky,mastodon}.ts`) behind a small interface — Bluesky via the public AppView (`public.api.bsky.app`), Mastodon via instance REST. `instance_url` gated by `isSafeUrl()` + the existing SSRF/private-IP guard before any outbound request. 3. `services/watchWorker.ts` — BullMQ repeatable job (mirrors `federationSyncWorker`), interval `watch.pollIntervalMinutes` (default 10). 4. `addSocialPostAsEntry()` shared helper calling `checkDedup()` before insert; watches CRUD, `discovered-posts/:id/add|dismiss`, and the live direct-search endpoint. 5. Frontend: **Watch** nav item + page (Watches / Queue / Search tabs). Note: #117 (workers → separate process) is a **P2** prerequisite, not P1 — the single-process model is fine for polling, so P1 proceeds as-is.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
renovate/altcha-lib-2.x
renovate/altcha-3.x
deps/wave-1-low-risk
v9.4.0
v9.3.0
v9.2.0
v9.1.0
v9.0.0
v8.0.0
v7.3.0
v7.2.1
v7.2.0
v7.1.0
v7.0.0
v6.2.0
v6.1.0
v6.0.0
v5.0.1
v5.0.0
v4.0.0
v3.0.0
v2.0
v1.0.0
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
future
Unscoped / not yet prioritized — no milestone

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
future
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/TeaLeaves#22
No description provided.