[Backlog] AI provider abstraction and per-campaign BYO-AI #128
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context / Motivation
Whisper and LLM configuration are instance-global admin settings today:
get_whisper_config(webapp/backend/app/services/settings_service.py:161, keywhisper_config→WhisperConfig(endpoint_url, api_key)) andget_llm_config(:182, keyllm_config→LLMConfig(endpoint_url, api_key, model)), both single-rowapp_settingskeys with no per-campaign scoping. Transcription additionally assumes a specific custom endpoint shape:process_audioPOSTs the whole session to{whisper}/transcribe/session(tasks/reminder_tasks.py:1605).Value for self-hosters today: one instance, several groups, each bringing their own API key and paying for their own transcription/summarization.
Spec (starting sketch)
Provider interface — transcription: a small strategy layer with implementations for:
/transcribe/session, per-speaker multi-file — the richest contract; keep as default)/v1/audio/transcriptions, one file per request — adapter loops per-speaker tracks and merges via the existingmerge_attributed_transcript,services/audio_service.py:401)Provider interface — summarization/LLM: verify and state current shape —
LLMConfig(endpoint_url, api_key, model)already targets OpenAI-compatible chat endpoints generically (used acrossaudio_serviceand the lore/journal generators inreminder_tasks.py), so this side likely needs only config plumbing, not an adapter layer. Confirm during implementation and document.Per-campaign override: nullable campaign-level AI config (whisper endpoint+key, LLM endpoint+key+model) — falling back to instance defaults when unset. Keys encrypted at rest exactly like the existing settings secrets: the
_ENCRYPTED_KEYSAES-256-GCM pattern (settings_service.py:41-47+app/crypto.py, HKDF fromSECRET_KEY) — reuseencrypt_setting/decrypt_settingfor the campaign columns rather than inventing a second scheme. Same SSRF guard (normalize_service_url) on campaign-supplied URLs. GM-editable in campaign settings behind an instance-admin toggle ("allow campaign AI overrides").Usage counters: per-campaign counters (sessions transcribed, audio seconds, LLM tokens where reported) — groundwork for any future hosted metering; a simple counters table incremented from the pipeline tasks, surfaced read-only in campaign settings.
Diarization note (decided — do not revisit): provider-specific diarization is irrelevant here. Per-speaker Discord tracks make diarization unnecessary — speaker identity comes from the track, not the model. No adapter should grow a diarization option.
Out of scope
Open questions
Acceptance criteria (for when this is pulled)
References
webapp/backend/app/services/settings_service.py:161(get_whisper_config),:182(get_llm_config),:41-47(_ENCRYPTED_KEYS)webapp/backend/app/crypto.py(encrypt_setting/decrypt_setting, AES-256-GCM + HKDF)webapp/backend/app/tasks/reminder_tasks.py:1605(/transcribe/sessioncall insideprocess_audio)webapp/backend/app/services/audio_service.py:274/:329/:401(transcribe + merge contract)Filed from the July 2026 full-project review.
Progress, since the milestone absorbed this issue:
AsrProvidercontract (#350), the bundled WhisperX adapter, and the OpenAI-compatible adapter covering OpenAI and Groq (PR #496) — one file per request, per-speaker tracks looped and merged by the existing caller code, the 25 MB cap honoured by caller-side chunk-and-remap. Diarization stays out, as decided.encrypt_settingscheme, the same SSRF guard on campaign URLs, GM-editable behind an instance-admin toggle, falling back to instance defaults. Being built after the #352 client migration lands so the Alembic chain stays linear. Will report here when it lands.The per-campaign half landed in PR #501 (merged 2026-09-05, CI green; 2185 backend / 503 frontend tests), which completes this issue as absorbed into v4.2.0.
d8e9fa0b1c2d:campaign_ai_settings— per campaign, each side's endpoint, key (encrypted with the existing AES-256-GCM/HKDF scheme), model, provider and window.allow_campaign_ai_overrides(default off) in Admin → Bot Settings; turning it off makes every override inert at once without deleting anything.@llm_taskdecorator, planning, Workbench,/ask, lore and beat services); admin test buttons, bot startup and the canary stay instance-level by design; an AST guard test keeps future call sites honest./api/campaigns/{id}/ai-settingswith test buttons; keys masked, never returned or logged; SSRF guard shared with instance settings.CampaignAiSettingson the campaign settings page when the switch is on.Original acceptance criteria: pipeline unchanged on the bundled endpoint (regression suite green); OpenAI-compatible provider through the per-speaker fixture via the conformance suite and wire fakes (#496 — Groq shares the adapter; no separate chunking adapter was needed because the caller-side span cutting already honours the 25 MB cap); campaign with its own key transcribes via its provider and falls back when unset (precedence tests); keys never in responses or logs (masking tests); counters increment per processed session (#357's usage rows, richer than counters).
Open questions from the original body, answered: rate/failure isolation per campaign is handled by #356's per-provider slots (a dead campaign endpoint queues only its own sessions) — no circuit breaker yet; overrides apply to the whole pipeline including lore, not only transcription and summary.