feat: at-least-once bot event delivery via Redis Stream + idempotency (#81) #166

Merged

claude-bot merged 1 commit from feat/81-stream-bot-events into main

2026-07-15 23:49:53 +00:00

claude-bot commented

2026-07-15 23:44:51 +00:00

Contributor

Summary

Backend→bot event delivery was split across mechanisms with different reliability. Pub/sub-only events (session_summarised, session_completed, vote_update, session_summary_approved) were silently dropped whenever the bot was down/restarting, and the HTTP /notify fast path could double-post on a slow response. This unifies all events onto a durable Redis Stream with idempotency. Cross-component, one PR.

Design

Transport: backend XADDs every event to qb:bot:events (Redis DB 0, MAXLEN ~10000); bot consumes via a consumer group qb-bot-workers (XREADGROUP), created with MKSTREAM at id 0 so events published before the group existed still deliver.
At-least-once: on start/reconnect the bot drains its pending backlog (XREADGROUP … 0) then blocks for new entries (… >); XACK happens after the handler runs. No polling.
Idempotency: every event is stamped with a UUID event_id at publish time. on_bot_notify does check → handle → mark against qb:bot:handled:<event_id> (TTL 1h, Redis with in-memory fallback). Mark-before-ack, so a crash between mark and ack redelivers and is skipped — no double post.
/notify fast path retained but now stamps the same event_id before posting, so the stream copy dedupes against it — closes the original double-post.
Recording live-status (recording:{guild_id}) stays plain pub/sub (ephemeral, fine to lose).

Files

Backend: services/bot_pubsub.py (XADD + stamp_event_id), tasks/reminder_tasks.py (stamp id before /notify).
Bot: services/redis_bus.py (consumer-group reader: _ensure_group/_drain_pending/_process_entry), cogs/notifications.py (event_id idempotency).
Tests: tests/test_bot_pubsub.py (new), bot/tests/test_redis_bus.py (new), bot/tests/test_notifications.py (+4).

Verification

Full suites green on the test DB: backend 367 passed, bot 179 passed; ruff 0.4.4 format+check clean on changed backend files. A real-Redis cross-restart integration test isn't feasible in the mock harness; a step-by-step manual verification procedure (event published while bot stopped → delivered once on restart; duplicate event_id skipped; crash-recovery; recording channel unchanged) is documented and will be added as a PR comment.

Closes #81

🤖 Generated with Claude Code

## Summary Backend→bot event delivery was split across mechanisms with different reliability. Pub/sub-only events (`session_summarised`, `session_completed`, `vote_update`, `session_summary_approved`) were **silently dropped** whenever the bot was down/restarting, and the HTTP `/notify` fast path could **double-post** on a slow response. This unifies all events onto a durable Redis Stream with idempotency. Cross-component, one PR. ## Design - **Transport**: backend `XADD`s every event to `qb:bot:events` (Redis DB 0, `MAXLEN ~10000`); bot consumes via a consumer group `qb-bot-workers` (`XREADGROUP`), created with `MKSTREAM` at id `0` so events published before the group existed still deliver. - **At-least-once**: on start/reconnect the bot drains its pending backlog (`XREADGROUP … 0`) then blocks for new entries (`… >`); `XACK` happens **after** the handler runs. No polling. - **Idempotency**: every event is stamped with a UUID `event_id` at publish time. `on_bot_notify` does **check → handle → mark** against `qb:bot:handled:<event_id>` (TTL 1h, Redis with in-memory fallback). Mark-before-ack, so a crash between mark and ack redelivers and is skipped — no double post. - **`/notify` fast path retained** but now stamps the same `event_id` before posting, so the stream copy dedupes against it — closes the original double-post. - **Recording live-status** (`recording:{guild_id}`) stays plain pub/sub (ephemeral, fine to lose). ## Files - Backend: `services/bot_pubsub.py` (XADD + `stamp_event_id`), `tasks/reminder_tasks.py` (stamp id before `/notify`). - Bot: `services/redis_bus.py` (consumer-group reader: `_ensure_group`/`_drain_pending`/`_process_entry`), `cogs/notifications.py` (event_id idempotency). - Tests: `tests/test_bot_pubsub.py` (new), `bot/tests/test_redis_bus.py` (new), `bot/tests/test_notifications.py` (+4). ## Verification Full suites green on the test DB: **backend 367 passed, bot 179 passed**; ruff 0.4.4 format+check clean on changed backend files. A real-Redis cross-restart integration test isn't feasible in the mock harness; a step-by-step manual verification procedure (event published while bot stopped → delivered once on restart; duplicate `event_id` skipped; crash-recovery; recording channel unchanged) is documented and will be added as a PR comment. Closes #81 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit

2026-07-15 23:44:51 +00:00

feat: unify bot event delivery over a Redis Stream with idempotency

CI / Docker image build (pull_request) Successful in 16s

Details

CI / Backend lint (ruff) (pull_request) Successful in 1m37s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 1m44s

Details

CI / Bot tests and audit (pull_request) Successful in 3m56s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 4m57s

Details

d46df5beb2

Replace the split pub/sub-vs-HTTP transport for backend->bot session events
with a single durable Redis Stream (qb:bot:events) consumed through a consumer
group, giving at-least-once delivery instead of fire-and-forget pub/sub that
silently dropped events fired while the bot was down.

Backend (webapp):
- bot_pubsub.publish_bot_event / publish_bot_event_async now XADD to the stream
  (MAXLEN ~10000) and stamp a UUID event_id via stamp_event_id() instead of
  PUBLISHing to qb:bot:notifications. All event types (session_confirmed,
  session_reminder, session_completed, session_summarised, vote_update,
  session_cancelled, session_summary_approved, session_proposed,
  session_vote_reminder, test) flow through the stream.
- The HTTP /notify fast path for session_proposed and session_vote_reminder
  now stamps the shared event_id before posting, so the stream fallback copy
  carries the same id and the bot dedupes it (fixes the slow-response
  double-post).
- Recording live-status stays plain pub/sub (ephemeral, fine to lose).

Bot:
- redis_bus.listen_for_bot_notifications consumes via XREADGROUP: create the
  group with MKSTREAM (ignore BUSYGROUP), drain the delivered-but-unacked
  backlog at id 0 on start, then BLOCK for new entries at >. XACK only after
  the handler returns.
- notifications.on_bot_notify dedupes on event_id against a short-TTL (1h)
  Redis store (qb:bot:handled:*) with an in-memory fallback, shared by the
  stream and /notify paths. Mark-before-ack: a crash between handling and ack
  redelivers the entry, which the store then skips instead of double-posting.

Tests: publish stamps event_id + XADDs; four previously pub/sub-only types now
publish to the stream; bot idempotency skips duplicate event_ids; stream
consume+ack, backlog drain, and BUSYGROUP handling. Backend 367 pass, bot 179.

Closes #81

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed

2026-07-15 23:45:00 +00:00

claude-bot commented

2026-07-15 23:45:06 +00:00

Author

Contributor

Manual verification (real Redis, cross-restart at-least-once)

The mock test harness can't exercise a real Redis stream across a bot restart; run this against a live dev stack (Redis DB 0):

Bring up backend + Redis + bot.
Stop only the bot: docker compose stop bot.
Publish an event with the backend up — e.g. cancel a bot-connected session, or in a backend shell:
from app.services.bot_pubsub import publish_bot_event; publish_bot_event({"event_type":"session_cancelled","session_id":"…","guild_id":"…","channel_id":"<real channel>","extra":{"title":"T","campaign_name":"C"}})
Confirm it queued: redis-cli -n 0 XLEN qb:bot:events increments; no Discord post yet.
docker compose start bot → expect exactly one embed (backlog drain), and XPENDING qb:bot:events qb-bot-workers → 0 after handling.
Duplicate: publish again reusing the same event_id (or POST it to /notify and XADD it) → one post; bot debug-logs Skipping already-handled event <id>; TTL qb:bot:handled:<event_id> ≈ 3600.
Crash recovery: publish, kill the bot mid-handle (before ack), restart → entry redelivered from the PEL and skipped (marked before ack), no repost.
Recording status unchanged: start a recording → live status still flows over recording:{guild_id} pub/sub, not the stream.

### Manual verification (real Redis, cross-restart at-least-once) The mock test harness can't exercise a real Redis stream across a bot restart; run this against a live dev stack (Redis DB 0): 1. Bring up backend + Redis + bot. 2. Stop **only** the bot: `docker compose stop bot`. 3. Publish an event with the backend up — e.g. cancel a bot-connected session, or in a backend shell: `from app.services.bot_pubsub import publish_bot_event; publish_bot_event({"event_type":"session_cancelled","session_id":"…","guild_id":"…","channel_id":"<real channel>","extra":{"title":"T","campaign_name":"C"}})` Confirm it queued: `redis-cli -n 0 XLEN qb:bot:events` increments; no Discord post yet. 4. `docker compose start bot` → expect **exactly one** embed (backlog drain), and `XPENDING qb:bot:events qb-bot-workers` → 0 after handling. 5. **Duplicate**: publish again reusing the same `event_id` (or POST it to `/notify` *and* XADD it) → one post; bot debug-logs `Skipping already-handled event <id>`; `TTL qb:bot:handled:<event_id>` ≈ 3600. 6. **Crash recovery**: publish, kill the bot mid-handle (before ack), restart → entry redelivered from the PEL and skipped (marked before ack), no repost. 7. **Recording status unchanged**: start a recording → live status still flows over `recording:{guild_id}` pub/sub, not the stream.

claude-bot merged commit fda7c53a7c into main

2026-07-15 23:49:53 +00:00

claude-bot deleted branch feat/81-stream-bot-events

2026-07-15 23:49:53 +00:00

claude-bot referenced this pull request from a commit

2026-07-15 23:49:54 +00:00

Merge pull request 'feat: at-least-once bot event delivery via Redis Stream + idempotency (#81)' (#166) from feat/81-stream-bot-events into main

claude-bot referenced this pull request

2026-07-16 02:14:36 +00:00

[Ops] Live-stack verification of v3.4.0 reliability changes #175