[Backlog] AI provider abstraction and per-campaign BYO-AI

claude-bot commented

2026-07-14 19:58:08 +00:00

Collaborator

Context / Motivation

Whisper and LLM configuration are instance-global admin settings today: get_whisper_config (webapp/backend/app/services/settings_service.py:161, key whisper_config → WhisperConfig(endpoint_url, api_key)) and get_llm_config (:182, key llm_config → LLMConfig(endpoint_url, api_key, model)), both single-row app_settings keys with no per-campaign scoping. Transcription additionally assumes a specific custom endpoint shape: process_audio POSTs the whole session to {whisper}/transcribe/session (tasks/reminder_tasks.py:1605).

Value for self-hosters today: one instance, several groups, each bringing their own API key and paying for their own transcription/summarization.

Spec (starting sketch)

Provider interface — transcription: a small strategy layer with implementations for:

current custom endpoint (/transcribe/session, per-speaker multi-file — the richest contract; keep as default)
OpenAI-compatible audio API (/v1/audio/transcriptions, one file per request — adapter loops per-speaker tracks and merges via the existing merge_attributed_transcript, services/audio_service.py:401)
Groq (OpenAI-shaped but 25 MB per-request cap — adapter must chunk long tracks and stitch segments with time offsets)

Provider interface — summarization/LLM: verify and state current shape — LLMConfig(endpoint_url, api_key, model) already targets OpenAI-compatible chat endpoints generically (used across audio_service and the lore/journal generators in reminder_tasks.py), so this side likely needs only config plumbing, not an adapter layer. Confirm during implementation and document.

Per-campaign override: nullable campaign-level AI config (whisper endpoint+key, LLM endpoint+key+model) — falling back to instance defaults when unset. Keys encrypted at rest exactly like the existing settings secrets: the _ENCRYPTED_KEYS AES-256-GCM pattern (settings_service.py:41-47 + app/crypto.py, HKDF from SECRET_KEY) — reuse encrypt_setting/decrypt_setting for the campaign columns rather than inventing a second scheme. Same SSRF guard (normalize_service_url) on campaign-supplied URLs. GM-editable in campaign settings behind an instance-admin toggle ("allow campaign AI overrides").

Usage counters: per-campaign counters (sessions transcribed, audio seconds, LLM tokens where reported) — groundwork for any future hosted metering; a simple counters table incremented from the pipeline tasks, surfaced read-only in campaign settings.

Diarization note (decided — do not revisit): provider-specific diarization is irrelevant here. Per-speaker Discord tracks make diarization unnecessary — speaker identity comes from the track, not the model. No adapter should grow a diarization option.

Out of scope

Hosted billing/metering (counters only).
New AI features; this is plumbing.
UI key-vault management beyond set/clear/masked-display.

Open questions

Rate/failure isolation per campaign: does one campaign's dead endpoint need circuit-breaking so Beat/queue throughput isn't consumed by its retries?
Should per-campaign overrides apply to the lore pipeline too, or transcription+summary only at first?
Groq chunking: silence-split vs fixed-window with overlap (pick during implementation; note transcript stitching accuracy).

Acceptance criteria (for when this is pulled)

Pipeline runs unchanged with the current custom endpoint (regression).
OpenAI-compatible and Groq providers pass a per-speaker fixture through to a correct merged transcript (mock servers).
Campaign with its own key transcribes via its provider; unset falls back to instance config; campaign keys never appear in API responses or logs.
Counters increment once per processed session.

References

webapp/backend/app/services/settings_service.py:161 (get_whisper_config), :182 (get_llm_config), :41-47 (_ENCRYPTED_KEYS)
webapp/backend/app/crypto.py (encrypt_setting/decrypt_setting, AES-256-GCM + HKDF)
webapp/backend/app/tasks/reminder_tasks.py:1605 (/transcribe/session call inside process_audio)
webapp/backend/app/services/audio_service.py:274/:329/:401 (transcribe + merge contract)

Filed from the July 2026 full-project review.

## Context / Motivation Whisper and LLM configuration are **instance-global admin settings** today: `get_whisper_config` (`webapp/backend/app/services/settings_service.py:161`, key `whisper_config` → `WhisperConfig(endpoint_url, api_key)`) and `get_llm_config` (`:182`, key `llm_config` → `LLMConfig(endpoint_url, api_key, model)`), both single-row `app_settings` keys with no per-campaign scoping. Transcription additionally assumes a specific custom endpoint shape: `process_audio` POSTs the whole session to `{whisper}/transcribe/session` (`tasks/reminder_tasks.py:1605`). Value for self-hosters **today**: one instance, several groups, each bringing their own API key and paying for their own transcription/summarization. ## Spec (starting sketch) **Provider interface — transcription**: a small strategy layer with implementations for: - current custom endpoint (`/transcribe/session`, per-speaker multi-file — the richest contract; keep as default) - OpenAI-compatible audio API (`/v1/audio/transcriptions`, one file per request — adapter loops per-speaker tracks and merges via the existing `merge_attributed_transcript`, `services/audio_service.py:401`) - Groq (OpenAI-shaped but 25 MB per-request cap — adapter must chunk long tracks and stitch segments with time offsets) **Provider interface — summarization/LLM**: verify and state current shape — `LLMConfig(endpoint_url, api_key, model)` already targets OpenAI-compatible chat endpoints generically (used across `audio_service` and the lore/journal generators in `reminder_tasks.py`), so this side likely needs only config plumbing, not an adapter layer. Confirm during implementation and document. **Per-campaign override**: nullable campaign-level AI config (whisper endpoint+key, LLM endpoint+key+model) — falling back to instance defaults when unset. Keys encrypted at rest **exactly like the existing settings secrets**: the `_ENCRYPTED_KEYS` AES-256-GCM pattern (`settings_service.py:41-47` + `app/crypto.py`, HKDF from `SECRET_KEY`) — reuse `encrypt_setting`/`decrypt_setting` for the campaign columns rather than inventing a second scheme. Same SSRF guard (`normalize_service_url`) on campaign-supplied URLs. GM-editable in campaign settings behind an instance-admin toggle ("allow campaign AI overrides"). **Usage counters**: per-campaign counters (sessions transcribed, audio seconds, LLM tokens where reported) — groundwork for any future hosted metering; a simple counters table incremented from the pipeline tasks, surfaced read-only in campaign settings. **Diarization note (decided — do not revisit)**: provider-specific diarization is irrelevant here. Per-speaker Discord tracks make diarization unnecessary — speaker identity comes from the track, not the model. No adapter should grow a diarization option. ## Out of scope - Hosted billing/metering (counters only). - New AI features; this is plumbing. - UI key-vault management beyond set/clear/masked-display. ## Open questions - Rate/failure isolation per campaign: does one campaign's dead endpoint need circuit-breaking so Beat/queue throughput isn't consumed by its retries? - Should per-campaign overrides apply to the lore pipeline too, or transcription+summary only at first? - Groq chunking: silence-split vs fixed-window with overlap (pick during implementation; note transcript stitching accuracy). ## Acceptance criteria (for when this is pulled) - Pipeline runs unchanged with the current custom endpoint (regression). - OpenAI-compatible and Groq providers pass a per-speaker fixture through to a correct merged transcript (mock servers). - Campaign with its own key transcribes via its provider; unset falls back to instance config; campaign keys never appear in API responses or logs. - Counters increment once per processed session. ## References - `webapp/backend/app/services/settings_service.py:161` (`get_whisper_config`), `:182` (`get_llm_config`), `:41-47` (`_ENCRYPTED_KEYS`) - `webapp/backend/app/crypto.py` (`encrypt_setting`/`decrypt_setting`, AES-256-GCM + HKDF) - `webapp/backend/app/tasks/reminder_tasks.py:1605` (`/transcribe/session` call inside `process_audio`) - `webapp/backend/app/services/audio_service.py:274/:329/:401` (transcribe + merge contract) _Filed from the July 2026 full-project review._

claude-bot added the

backend

enhancement

recording

labels

2026-07-14 19:58:33 +00:00

claude-bot added this to the Backlog - Speculative / Long-Term milestone

2026-07-14 19:58:34 +00:00

claude-bot referenced this issue

2026-08-25 19:18:25 +00:00

[Review] Full audit of the session lifecycle: prep, recording, transcription, summarization, lore — scoping v4.0.0 #319

claude-bot referenced this issue

2026-08-25 19:32:48 +00:00

[Review] Full audit of the session lifecycle: prep, recording, transcription, summarization, lore — scoping v4.0.0 #319

claude-bot modified the milestone from Backlog - Speculative / Long-Term to v4.2.0 - AI Provider Abstraction & BYO-AI

2026-08-25 20:39:57 +00:00

claude-bot referenced this issue

2026-08-25 20:45:22 +00:00

[Review] Full audit of the session lifecycle: prep, recording, transcription, summarization, lore — scoping v4.0.0 #319

claude-bot referenced this issue

2026-09-05 03:41:37 +00:00

feat: OpenAI-compatible transcription adapter and an ASR provider select (#350, #128) #496

claude-bot referenced this issue from a commit

2026-09-05 03:47:46 +00:00

Merge PR #496: OpenAI-compatible transcription adapter and ASR provider select (#350, #128)

claude-bot commented

2026-09-05 03:48:21 +00:00

Author

Collaborator

Progress, since the milestone absorbed this issue:

Provider interface — transcription: done. AsrProvider contract (#350), the bundled WhisperX adapter, and the OpenAI-compatible adapter covering OpenAI and Groq (PR #496) — one file per request, per-speaker tracks looped and merged by the existing caller code, the 25 MB cap honoured by caller-side chunk-and-remap. Diarization stays out, as decided.
Provider interface — LLM: done. Confirmed as the original spec suspected: config plumbing plus a provider object, not a transport rewrite (#351, #484).
Usage counters: done, and more than counters — per-session usage and cost rows with per-campaign rollups (#357).
Per-campaign override (BYO-AI): next. Nullable campaign-level ASR/LLM config, keys encrypted with the existing encrypt_setting scheme, the same SSRF guard on campaign URLs, GM-editable behind an instance-admin toggle, falling back to instance defaults. Being built after the #352 client migration lands so the Alembic chain stays linear. Will report here when it lands.

Progress, since the milestone absorbed this issue: - **Provider interface — transcription: done.** `AsrProvider` contract (#350), the bundled WhisperX adapter, and the OpenAI-compatible adapter covering OpenAI and Groq (PR #496) — one file per request, per-speaker tracks looped and merged by the existing caller code, the 25 MB cap honoured by caller-side chunk-and-remap. Diarization stays out, as decided. - **Provider interface — LLM: done.** Confirmed as the original spec suspected: config plumbing plus a provider object, not a transport rewrite (#351, #484). - **Usage counters: done, and more than counters** — per-session usage and cost rows with per-campaign rollups (#357). - **Per-campaign override (BYO-AI): next.** Nullable campaign-level ASR/LLM config, keys encrypted with the existing `encrypt_setting` scheme, the same SSRF guard on campaign URLs, GM-editable behind an instance-admin toggle, falling back to instance defaults. Being built after the #352 client migration lands so the Alembic chain stays linear. Will report here when it lands.

claude-bot referenced this issue

2026-09-05 04:27:28 +00:00

[Backend] Evaluate ASR and LLM providers against the golden corpus before committing #360

claude-bot referenced this issue

2026-09-05 04:37:55 +00:00

[Backend] Evaluate ASR and LLM providers against the golden corpus before committing #360

rbrooks referenced this issue from a commit

2026-09-05 05:15:16 +00:00

db: campaign_ai_settings, where a group's own AI providers live (#128)

rbrooks referenced this issue from a commit

2026-09-05 05:15:16 +00:00

feat(settings): per-campaign AI configuration with instance fallback (#128)

rbrooks referenced this issue from a commit

2026-09-05 05:15:16 +00:00

feat(tasks): resolve the campaign's providers at every origin (#128)

rbrooks referenced this issue from a commit

2026-09-05 05:15:16 +00:00

feat(api): a GM's campaign AI settings, and the admin switch above them (#128)

rbrooks referenced this issue from a commit

2026-09-05 05:15:16 +00:00

feat(frontend): a campaign's own AI providers, and the admin switch (#128)

rbrooks referenced this issue from a commit

2026-09-05 05:15:16 +00:00

test: precedence, secrecy, and a guard for the call sites nobody has written yet (#128)

rbrooks referenced this issue from a commit

2026-09-05 05:15:16 +00:00

docs: what an operator is agreeing to when they let campaigns bring their own AI (#128)

rbrooks referenced this issue from a commit

2026-09-05 05:15:16 +00:00

db: chain campaign_ai_settings behind the transcript-words migration (#128)

rbrooks referenced this issue from a commit

2026-09-05 05:15:16 +00:00

test: await the gap derivation, which became a coroutine under us (#128, #352)

claude-bot referenced this issue from a pull request that will close it,

2026-09-05 05:15:42 +00:00

feat: per-campaign AI providers — a campaign brings its own ASR and LLM, behind an instance switch (#128) #501

claude-bot referenced this issue

2026-09-05 05:16:29 +00:00

[Backend] Follow-ups from the provider evaluation: usage rows misclassify sniffed managed runs, managed ASR is unpriceable, and the live tooling cannot reach the managed adapter (#360) #502

claude-bot closed this issue

2026-09-05 05:24:08 +00:00

claude-bot referenced this issue from a commit

2026-09-05 05:24:09 +00:00

Merge PR #501: per-campaign AI providers behind an instance switch (#128)

claude-bot commented

2026-09-05 05:25:03 +00:00

Author

Collaborator

The per-campaign half landed in PR #501 (merged 2026-09-05, CI green; 2185 backend / 503 frontend tests), which completes this issue as absorbed into v4.2.0.

Migration d8e9fa0b1c2d: campaign_ai_settings — per campaign, each side's endpoint, key (encrypted with the existing AES-256-GCM/HKDF scheme), model, provider and window.
Instance switch allow_campaign_ai_overrides (default off) in Admin → Bot Settings; turning it off makes every override inert at once without deleting anything.
One resolver per side with the precedence documented next to the existing not-configured policies: no campaign → instance; no row or blank endpoint → instance; switch off → instance; otherwise the campaign's, each side independently. An override that is in effect never silently falls back to the admin's provider — a rejected campaign URL or an undecryptable key is a visible failure naming the campaign's settings, because substituting the instance's provider would send a group's audio somewhere they did not choose.
Every origin with a campaign in scope passes it (the audio task, the @llm_task decorator, planning, Workbench, /ask, lore and beat services); admin test buttons, bot startup and the canary stay instance-level by design; an AST guard test keeps future call sites honest.
GM API under /api/campaigns/{id}/ai-settings with test buttons; keys masked, never returned or logged; SSRF guard shared with instance settings.
CampaignAiSettings on the campaign settings page when the switch is on.

Original acceptance criteria: pipeline unchanged on the bundled endpoint (regression suite green); OpenAI-compatible provider through the per-speaker fixture via the conformance suite and wire fakes (#496 — Groq shares the adapter; no separate chunking adapter was needed because the caller-side span cutting already honours the 25 MB cap); campaign with its own key transcribes via its provider and falls back when unset (precedence tests); keys never in responses or logs (masking tests); counters increment per processed session (#357's usage rows, richer than counters).

Open questions from the original body, answered: rate/failure isolation per campaign is handled by #356's per-provider slots (a dead campaign endpoint queues only its own sessions) — no circuit breaker yet; overrides apply to the whole pipeline including lore, not only transcription and summary.

The per-campaign half landed in **PR #501** (merged 2026-09-05, CI green; 2185 backend / 503 frontend tests), which completes this issue as absorbed into v4.2.0. - Migration `d8e9fa0b1c2d`: `campaign_ai_settings` — per campaign, each side's endpoint, key (encrypted with the existing AES-256-GCM/HKDF scheme), model, provider and window. - Instance switch `allow_campaign_ai_overrides` (default off) in Admin → Bot Settings; turning it off makes every override inert at once without deleting anything. - One resolver per side with the precedence documented next to the existing not-configured policies: no campaign → instance; no row or blank endpoint → instance; switch off → instance; otherwise the campaign's, each side independently. An override that is in effect never silently falls back to the admin's provider — a rejected campaign URL or an undecryptable key is a visible failure naming the campaign's settings, because substituting the instance's provider would send a group's audio somewhere they did not choose. - Every origin with a campaign in scope passes it (the audio task, the `@llm_task` decorator, planning, Workbench, `/ask`, lore and beat services); admin test buttons, bot startup and the canary stay instance-level by design; an AST guard test keeps future call sites honest. - GM API under `/api/campaigns/{id}/ai-settings` with test buttons; keys masked, never returned or logged; SSRF guard shared with instance settings. - `CampaignAiSettings` on the campaign settings page when the switch is on. Original acceptance criteria: pipeline unchanged on the bundled endpoint (regression suite green); OpenAI-compatible provider through the per-speaker fixture via the conformance suite and wire fakes (#496 — Groq shares the adapter; no separate chunking adapter was needed because the caller-side span cutting already honours the 25 MB cap); campaign with its own key transcribes via its provider and falls back when unset (precedence tests); keys never in responses or logs (masking tests); counters increment per processed session (#357's usage rows, richer than counters). Open questions from the original body, answered: rate/failure isolation per campaign is handled by #356's per-provider slots (a dead campaign endpoint queues only its own sessions) — no circuit breaker yet; overrides apply to the whole pipeline including lore, not only transcription and summary.

rbrooks referenced this issue from a commit

2026-09-05 05:41:26 +00:00

db: chain the usage-identity columns behind #501's campaign settings (#502)

claude-bot referenced this issue

2026-09-05 06:04:34 +00:00

chore(release): v4.2.0 — AI Provider Abstraction & BYO-AI #504