[Bot] Make the bot event stream's trim limit configurable for hosted scale #415

New issue

Closed

opened 2026-08-25 20:44:56 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-08-25 20:44:56 +00:00

Contributor

Severity: LOW

Found in the August 2026 session lifecycle review (#319).

The Redis Stream carrying bot events between backend and bot is capped at roughly 10,000 entries; a bot that is disconnected or down long enough on a busy, multi-tenant hosted instance will have the oldest unconsumed events trimmed out from under it before it reconnects, losing them permanently rather than catching up on reconnect. The delivery mechanism itself (consumer group, per-event dedupe, XACK-after-handle, pending-entry drain on restart) is solid — this is purely a capacity/sizing question that was never revisited for hosted multi-tenant volume.

Evidence

webapp/backend/app/services/bot_pubsub.py:44 (BOT_EVENTS_MAXLEN = 10000), applied as the stream's approximate maxlen at :88 and :100.
No configuration surface exists to raise this value per-deployment; it is a hardcoded module constant.

Failure scenario
A hosted instance serving many tenants generates event volume well above what a single self-hosted instance would. The bot goes down for an extended maintenance window (or a slow redeploy). By the time it reconnects, more than 10,000 events have been produced across all tenants in the interim, and the oldest ones — for whichever tenants happened to generate events early in the outage — have already aged out of the stream and are lost, with no catch-up possible for those specific notifications.

Proposed fix
Make BOT_EVENTS_MAXLEN a configurable setting rather than a hardcoded constant, size it deliberately for the hosted deployment's expected multi-tenant event volume (with headroom for a realistic maintenance-window outage duration), and consider alerting if stream trimming is actually occurring in practice (Redis XINFO STREAM reports entries-added vs. current length, which can detect this).

Acceptance criteria

BOT_EVENTS_MAXLEN is configurable rather than hardcoded.
The hosted deployment profile sets a value sized for expected multi-tenant volume and outage duration.
Some observability exists to detect when the stream is actively trimming unconsumed entries.

**Severity: LOW** Found in the August 2026 session lifecycle review (#319). The Redis Stream carrying bot events between backend and bot is capped at roughly 10,000 entries; a bot that is disconnected or down long enough on a busy, multi-tenant hosted instance will have the oldest unconsumed events trimmed out from under it before it reconnects, losing them permanently rather than catching up on reconnect. The delivery mechanism itself (consumer group, per-event dedupe, XACK-after-handle, pending-entry drain on restart) is solid — this is purely a capacity/sizing question that was never revisited for hosted multi-tenant volume. **Evidence** - `webapp/backend/app/services/bot_pubsub.py:44` (`BOT_EVENTS_MAXLEN = 10000`), applied as the stream's approximate `maxlen` at `:88` and `:100`. - No configuration surface exists to raise this value per-deployment; it is a hardcoded module constant. **Failure scenario** A hosted instance serving many tenants generates event volume well above what a single self-hosted instance would. The bot goes down for an extended maintenance window (or a slow redeploy). By the time it reconnects, more than 10,000 events have been produced across all tenants in the interim, and the oldest ones — for whichever tenants happened to generate events early in the outage — have already aged out of the stream and are lost, with no catch-up possible for those specific notifications. **Proposed fix** Make `BOT_EVENTS_MAXLEN` a configurable setting rather than a hardcoded constant, size it deliberately for the hosted deployment's expected multi-tenant event volume (with headroom for a realistic maintenance-window outage duration), and consider alerting if stream trimming is actually occurring in practice (Redis `XINFO STREAM` reports entries-added vs. current length, which can detect this). **Acceptance criteria** - [ ] `BOT_EVENTS_MAXLEN` is configurable rather than hardcoded. - [ ] The hosted deployment profile sets a value sized for expected multi-tenant volume and outage duration. - [ ] Some observability exists to detect when the stream is actively trimming unconsumed entries.

claude-bot added this to the v4.1.0 - Data Durability & Recovery milestone

2026-08-25 20:44:56 +00:00

claude-bot added the

bot

enhancement

labels

2026-08-25 20:44:56 +00:00