[Backend] Provider-aware concurrency caps in the Celery queue #356
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found in the August 2026 session lifecycle review (#319).
Why
Session processing is bursty and highly correlated — most groups play Friday and Saturday evening, in a couple of timezone clusters. A single Friday night can deliver a large number of tracks into a narrow window.
Every provider has a limit, and they differ: managed APIs publish concurrency or rate limits, a self-hosted llama.cpp instance has a fixed slot count (the current deployment runs
--parallel 2), and a CPU deployment is effectively serial. Exceeding any of them produces rate-limit errors or queue collapse rather than graceful slowdown.Proposed fix
Give each provider adapter a declared concurrency limit and enforce it in the Celery layer, so work queues rather than fails. Prefer a shape where a burst costs latency, not failures — this is asynchronous batch work and nobody needs their recap within 30 seconds.
Surface queue depth and estimated wait so a GM whose session lands in a busy window sees "queued, about 20 minutes" rather than silence. That connects to the async-feedback work in v4.3.0.
Acceptance criteria
Landed in PR #495 (merged 2026-09-05, CI green; 1929 backend / 489 frontend tests).
None) with.envoverridesQB_LLM_CONCURRENCY_LIMIT/QB_ASR_CONCURRENCY_LIMIT.app/services/provider_slots.py: a Redis lease semaphore (DB 0) with TTL leases, admission-order ranking, rolling hold durations for the estimate, and fail-open if Redis is unreachable.process_audioacquires the ASR slot and re-queues itself (self.retry, 20–40 s jittered, budget ~2 h) when none is free; every LLM path goes through onerun_with_slotat the transport dispatch with a bounded in-task wait; request-path callers never wait (a request-scope contextvar set by middleware, so a browser cannot hang on the queue).Retry-After(clamped to 10 min). An ASR 429 re-queues the session rather than failing it, and writes no usage row. An LLM 429 after transcription still fails after three attempts, deliberately: re-running would repeat the paid-for ASR.GET /api/admin/ai/queue,processing_waiton the session response, and a "queued behind N, about M minutes" note on the session page.Not done: no separate retry counter for queue waits versus transport retries — they share Celery's budget, documented at the call site.