Provider-slot concurrency cap is silently skipped in the post-session fan-out: "Event loop is closed" #573

New issue

Closed

opened 2026-09-09 06:43:39 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-09-09 06:43:39 +00:00

Contributor

Seen on the dev replay of 2026-09-09. Every LLM call made by the post-session fan-out tasks (storyline, title suggestions, lore proposals, highlights) logged:

Could not reach Redis to take a llm slot on llamacpp/10.3.0.28:8090 (RuntimeError: Event loop is closed); running without a concurrency cap for this call.

So the #356 slot machinery, which exists to keep a self-hosted router from being hammered by parallel tasks, never applies to the fan-out. The message is the graceful-degradation path doing its job, but the condition it degrades on is a bug, not an outage: the Redis client used by provider_slots is bound to an asyncio loop that a previous asyncio.run() in the same Celery worker process has already closed, and the next task's loop cannot use it.

Fix. Create the slot client per call (or per event loop) rather than caching one on the module or worker process; the pattern already used elsewhere for async Redis in Celery tasks should apply. A test that runs two fan-out tasks in sequence in one process and asserts the second still takes a slot.

Related: #356 (the cap), #357 (usage telemetry, which saw the calls).

**Seen on the dev replay of 2026-09-09.** Every LLM call made by the post-session fan-out tasks (storyline, title suggestions, lore proposals, highlights) logged: ``` Could not reach Redis to take a llm slot on llamacpp/10.3.0.28:8090 (RuntimeError: Event loop is closed); running without a concurrency cap for this call. ``` So the #356 slot machinery, which exists to keep a self-hosted router from being hammered by parallel tasks, never applies to the fan-out. The message is the graceful-degradation path doing its job, but the condition it degrades on is a bug, not an outage: the Redis client used by `provider_slots` is bound to an asyncio loop that a previous `asyncio.run()` in the same Celery worker process has already closed, and the next task's loop cannot use it. **Fix.** Create the slot client per call (or per event loop) rather than caching one on the module or worker process; the pattern already used elsewhere for async Redis in Celery tasks should apply. A test that runs two fan-out tasks in sequence in one process and asserts the second still takes a slot. Related: #356 (the cap), #357 (usage telemetry, which saw the calls).

rbrooks referenced this issue from a commit

2026-09-09 17:13:42 +00:00

fix(backend): pool Redis per event loop, so the slot cap survives a Celery task (#573)

claude-bot referenced this issue from a pull request that will close it,

2026-09-09 17:13:53 +00:00

fix(backend): pool Redis per event loop, so the slot cap survives a Celery task (#573) #582

claude-bot commented

2026-09-09 17:13:56 +00:00

Author

Contributor

Fix is on its way in the PR opened alongside this comment. Two things the investigation added to the issue as filed:

The same pattern lived in bot_pubsub (and, harmlessly for now, auth/session). The bot_pubsub case matters: detect_unread_trim and publish_bot_event_async run under asyncio.run in worker tasks, so the "were bot-event stream entries trimmed unread?" check has been answering "no data" rather than "no loss" whenever it ran as a worker's second-or-later task. Fixed by the same per-loop pool.
Reproduced against a real Redis before the fix: thirty asyncio.run cycles in one process, the fallback fired on every other one. The alternation is the tell: the failure poisons the connection, redis-py drops it, the next loop opens a fresh one, and the loop after that finds it closed.

Fix is on its way in the PR opened alongside this comment. Two things the investigation added to the issue as filed: - The same pattern lived in `bot_pubsub` (and, harmlessly for now, `auth/session`). The `bot_pubsub` case matters: `detect_unread_trim` and `publish_bot_event_async` run under `asyncio.run` in worker tasks, so the "were bot-event stream entries trimmed unread?" check has been answering "no data" rather than "no loss" whenever it ran as a worker's second-or-later task. Fixed by the same per-loop pool. - Reproduced against a real Redis before the fix: thirty `asyncio.run` cycles in one process, the fallback fired on every other one. The alternation is the tell: the failure poisons the connection, redis-py drops it, the next loop opens a fresh one, and the loop after that finds it closed.

claude-bot closed this issue

2026-09-09 17:33:33 +00:00

claude-bot referenced this issue from a commit

2026-09-09 17:33:35 +00:00

Merge PR #582: fix(backend): pool Redis per event loop, so the slot cap survives a Celery task (#573)