rbrooks/Quest-Board

Watch

Fork 0

[Ops] Live-stack verification of v3.4.0 reliability changes #175

New issue

Open

opened 2026-07-16 02:14:35 +00:00 by claude-bot · 5 comments

claude-bot commented

2026-07-16 02:14:35 +00:00

Contributor

Context

Several v3.4.0 reliability changes have behaviours that cannot be exercised in the mock/unit test harness (no live Redis, no real Discord voice, no multi-container runtime). Each was shipped with a documented manual procedure; this issue tracks actually rehearsing them once on a live/staging stack before treating v3.4.0 as production-verified.

Checklist

1. At-least-once bot delivery — cross-restart (#81)

Follow the procedure in the PR #166 comment:

Bring up backend + Redis + bot; docker compose stop bot.
Publish an event (e.g. cancel a bot-connected session); confirm redis-cli -n 0 XLEN qb:bot:events increments and no Discord post yet.
docker compose start bot → exactly one embed posts (backlog drain); XPENDING qb:bot:events qb-bot-workers → 0.
Re-publish with the same event_id → skipped (debug log), no second post; TTL qb:bot:handled:<id> ≈ 3600.
Kill the bot mid-handle before ack, restart → redelivered entry skipped (no repost).
Confirm recording live-status still flows over recording:{guild_id} pub/sub.

2. Disk-streamed recording — memory + e2e (#82)

Record a 15–20 min multi-speaker session; sample VmRSS from /proc/<bot_pid>/status every 30s.
Expect RSS to plateau (was ~0.6–0.7 GB/speaker-hour before); raw WAVs under audio_temp grow instead.
/record stop → no RSS spike (old getvalue() copy is gone); {session_dir}/{user_id}.wav appear, *_raw.wav deleted.
Run a short session fully e2e → backend transcribes and the summary embed posts; ffprobe a converted WAV = 16 kHz mono s16 (unchanged format).

3. Healthchecks + AOF (#84)

Per the "Verifying Healthchecks and AOF Persistence" checklist in docs/OPERATIONS.md:

docker compose ps shows health for worker and beat; docker pause (or SIGSTOP) the worker → flips unhealthy within a few intervals.
Sessions survive docker compose restart redis with AOF; confirm appendonly yes and AOF files on disk.

4. VAD trim (#101)

On a staging stack, enable vad_trim_enabled in Admin → Bot Settings; process a recorded session.
Confirm the "VAD trimmed Xh -> Yh" log line, and that the merged transcript ordering/timestamps/speaker-attribution match an untrimmed run of the same audio.

5. Redis pool auth round-trip (#96)

With a live Redis, exercise a full login → session-cookie → authenticated request cycle and confirm sessions read/write correctly through the new module-level pool under concurrency (the unit tests inject the user directly and don't touch the pool).

Acceptance

Each item above rehearsed once on a live/staging stack; any deviation from the documented expected behaviour filed as a bug.

## Context Several v3.4.0 reliability changes have behaviours that **cannot be exercised in the mock/unit test harness** (no live Redis, no real Discord voice, no multi-container runtime). Each was shipped with a documented manual procedure; this issue tracks actually **rehearsing them once on a live/staging stack** before treating v3.4.0 as production-verified. ## Checklist ### 1. At-least-once bot delivery — cross-restart (#81) Follow the procedure in the PR #166 comment: - Bring up backend + Redis + bot; `docker compose stop bot`. - Publish an event (e.g. cancel a bot-connected session); confirm `redis-cli -n 0 XLEN qb:bot:events` increments and no Discord post yet. - `docker compose start bot` → **exactly one** embed posts (backlog drain); `XPENDING qb:bot:events qb-bot-workers` → 0. - Re-publish with the same `event_id` → skipped (debug log), no second post; `TTL qb:bot:handled:<id>` ≈ 3600. - Kill the bot mid-handle before ack, restart → redelivered entry skipped (no repost). - Confirm recording live-status still flows over `recording:{guild_id}` pub/sub. ### 2. Disk-streamed recording — memory + e2e (#82) - Record a 15–20 min multi-speaker session; sample `VmRSS` from `/proc/<bot_pid>/status` every 30s. - Expect RSS to **plateau** (was ~0.6–0.7 GB/speaker-hour before); raw WAVs under `audio_temp` grow instead. - `/record stop` → no RSS spike (old `getvalue()` copy is gone); `{session_dir}/{user_id}.wav` appear, `*_raw.wav` deleted. - Run a short session fully e2e → backend transcribes and the summary embed posts; `ffprobe` a converted WAV = 16 kHz mono s16 (unchanged format). ### 3. Healthchecks + AOF (#84) Per the "Verifying Healthchecks and AOF Persistence" checklist in `docs/OPERATIONS.md`: - `docker compose ps` shows health for worker and beat; `docker pause` (or SIGSTOP) the worker → flips unhealthy within a few intervals. - Sessions survive `docker compose restart redis` with AOF; confirm `appendonly yes` and AOF files on disk. ### 4. VAD trim (#101) - On a staging stack, enable `vad_trim_enabled` in Admin → Bot Settings; process a recorded session. - Confirm the `"VAD trimmed Xh -> Yh"` log line, and that the merged transcript ordering/timestamps/speaker-attribution match an untrimmed run of the same audio. ### 5. Redis pool auth round-trip (#96) - With a live Redis, exercise a full login → session-cookie → authenticated request cycle and confirm sessions read/write correctly through the new module-level pool under concurrency (the unit tests inject the user directly and don't touch the pool). ## Acceptance - Each item above rehearsed once on a live/staging stack; any deviation from the documented expected behaviour filed as a bug.

claude-bot added the

backend

bot

recording

labels

2026-07-16 02:15:22 +00:00

claude-bot commented

2026-07-16 18:44:27 +00:00

Author

Contributor

v3.4.1 cut — healthcheck deviation from item 3 (#84) fixed & deployed

Recording a deviation found while rehearsing item 3 (Healthchecks + AOF, #84) on the live stack, plus the patch release that resolved it.

Deviation (bug): the #84 frontend healthcheck used wget --spider http://localhost/. The nginx:alpine image's /etc/hosts maps localhost to IPv6 ::1 only, but nginx binds IPv4 0.0.0.0:80, so the probe got connection-refused and the frontend container flapped unhealthy even though nginx was serving fine. (backend/bot use Debian images where localhost → 127.0.0.1, so they were unaffected.)

Fix: probe http://127.0.0.1/ instead — commit 1051f19 (fix(compose): frontend healthcheck probes 127.0.0.1 not localhost).

Released as v3.4.1 (b5b53ff): bumped APP_VERSION + BOT_EXPECTED_APP_VERSION to 3.4.1 and recorded the fix in the changelog. No API contract change (BOT_CONTRACT_VERSION stays 1). Forgejo release published; the images job built/pushed the :3.4.1 images.

Deployed (build-from-source, git pull --ff-only → v3.4.1 → docker compose up -d --build):

	dev (`docker-test`, 10.1.1.14)	prod (`docker-host`, 10.1.1.7)
commit	`v3.4.1` (`b5b53ff`)	`v3.4.1` (`b5b53ff`)
backend `APP_VERSION`	3.4.1	3.4.1
bot `BOT_EXPECTED`	3.4.1	3.4.1
`/ready`	—	200
alembic	—	`e5f6g7h8i9j0` (head, no new migration)
frontend healthcheck	healthy	healthy (12h uptime — fix confirmed)
all containers	healthy	healthy

Item 3 healthcheck sub-check is now verified green on both stacks (worker/beat/frontend health all reporting correctly). The AOF-persistence half of item 3, and items 1/2/4/5, remain to be rehearsed — leaving this issue open.

Two observations to follow up separately (not filed yet):

Prod's compose is loading docker-compose.override.yml (the dev port-bindings overlay) as one of its project.config_files — pre-existing, not from this deploy, but worth confirming it's intended.
The tag push produced a fast (~9s) failing workflow run alongside the successful release/images run — looks like the same fast-fail-on-push class as #174; worth a glance.

## v3.4.1 cut — healthcheck deviation from item 3 (#84) fixed & deployed Recording a deviation found while rehearsing **item 3 (Healthchecks + AOF, #84)** on the live stack, plus the patch release that resolved it. **Deviation (bug):** the #84 frontend healthcheck used `wget --spider http://localhost/`. The `nginx:alpine` image's `/etc/hosts` maps `localhost` to IPv6 `::1` **only**, but nginx binds IPv4 `0.0.0.0:80`, so the probe got connection-refused and the frontend container flapped **unhealthy** even though nginx was serving fine. (backend/bot use Debian images where `localhost` → `127.0.0.1`, so they were unaffected.) **Fix:** probe `http://127.0.0.1/` instead — commit `1051f19` (`fix(compose): frontend healthcheck probes 127.0.0.1 not localhost`). **Released as v3.4.1** (`b5b53ff`): bumped `APP_VERSION` + `BOT_EXPECTED_APP_VERSION` to `3.4.1` and recorded the fix in the changelog. No API contract change (`BOT_CONTRACT_VERSION` stays 1). Forgejo release published; the `images` job built/pushed the `:3.4.1` images. **Deployed (build-from-source, `git pull --ff-only` → `v3.4.1` → `docker compose up -d --build`):** | | dev (`docker-test`, 10.1.1.14) | prod (`docker-host`, 10.1.1.7) | |---|---|---| | commit | `v3.4.1` (b5b53ff) | `v3.4.1` (b5b53ff) | | backend `APP_VERSION` | 3.4.1 | 3.4.1 | | bot `BOT_EXPECTED` | 3.4.1 | 3.4.1 | | `/ready` | — | 200 | | alembic | — | `e5f6g7h8i9j0` (head, no new migration) | | frontend healthcheck | healthy | healthy (12h uptime — fix confirmed) | | all containers | healthy | healthy | Item 3 healthcheck sub-check is now **verified green on both stacks** (worker/beat/frontend health all reporting correctly). The AOF-persistence half of item 3, and items 1/2/4/5, remain to be rehearsed — leaving this issue open. **Two observations to follow up separately (not filed yet):** 1. Prod's compose is loading `docker-compose.override.yml` (the dev port-bindings overlay) as one of its `project.config_files` — pre-existing, not from this deploy, but worth confirming it's intended. 2. The tag push produced a fast (~9s) failing workflow run alongside the successful release/images run — looks like the same fast-fail-on-push class as #174; worth a glance.

claude-bot commented

2026-07-17 02:45:45 +00:00

Author

Contributor

Unrelated to this issue's checklist, but noting it here since this is the live-verification thread and the v3.5.0 frontend work has now landed on main (d339c41, 11 commits past v3.4.1).

v3.5.0 — Frontend Platform & PWA is code-complete (milestone: 0 open / 4 closed):

Issue	PR	Result
#21 real react-hooks lint	#180	31 warnings (stub, nothing enforced) → 0/0 with real enforcement
#103 shared API client	#181	expired session now redirects to login instead of stranding the user
#105 route code splitting	#183	entry chunk 685.50 → 244.25 kB (172.36 → 78.32 gzip), −64%
#108 PWA scaffolding	#184	installable; SW precaches shell + all 23 route chunks; `/api` NetworkOnly

Frontend suite went 69 → 106 tests. Post-merge CI green on main for each.

Two things that need a human before a v3.5.0 release:

One manual Lighthouse pass. There is no Lighthouse or headless browser in the toolchain, so #108's "installable" criterion was verified by asserting every input Lighthouse checks (manifest fields served as application/manifest+json, 192/512 any + maskable icons, SW registration, 0 CSP-blocked inline scripts — all confirmed against a real frontend-prod image) rather than by running Lighthouse itself. Offline behaviour likewise comes from the generated precache manifest and strategies, not from toggling a real browser offline.
Placeholder icons. webapp/frontend/public/ currently holds generated QB-monogram art. Drop real art in with the same filenames — no code change needed.

Also filed #182 (adopt the React Compiler lint rules from eslint-plugin-react-hooks v7) in the Backlog milestone. Measured, not guessed: 12 findings (11 × set-state-in-effect, 1 × preserve-manual-memoization) across 7 files; the other 12 compiler rules are already clean. It's scoped and ready to pull into a frontend milestone whenever one exists.

Deployed state is unchanged — dev and prod are still on the v3.4.1 tag. None of the above is live yet.

Unrelated to this issue's checklist, but noting it here since this is the live-verification thread and the v3.5.0 frontend work has now landed on `main` (`d339c41`, 11 commits past `v3.4.1`). **v3.5.0 — Frontend Platform & PWA is code-complete** (milestone: 0 open / 4 closed): | Issue | PR | Result | |---|---|---| | #21 real react-hooks lint | #180 | 31 warnings (stub, nothing enforced) → **0/0 with real enforcement** | | #103 shared API client | #181 | expired session now redirects to login instead of stranding the user | | #105 route code splitting | #183 | entry chunk **685.50 → 244.25 kB** (172.36 → 78.32 gzip), −64% | | #108 PWA scaffolding | #184 | installable; SW precaches shell + all 23 route chunks; `/api` NetworkOnly | Frontend suite went **69 → 106 tests**. Post-merge CI green on `main` for each. **Two things that need a human before a v3.5.0 release:** 1. **One manual Lighthouse pass.** There is no Lighthouse or headless browser in the toolchain, so #108's "installable" criterion was verified by asserting every input Lighthouse checks (manifest fields served as `application/manifest+json`, 192/512 `any` + `maskable` icons, SW registration, 0 CSP-blocked inline scripts — all confirmed against a real `frontend-prod` image) rather than by running Lighthouse itself. Offline behaviour likewise comes from the generated precache manifest and strategies, not from toggling a real browser offline. 2. **Placeholder icons.** `webapp/frontend/public/` currently holds generated QB-monogram art. Drop real art in with the same filenames — no code change needed. Also filed **#182** (adopt the React Compiler lint rules from eslint-plugin-react-hooks v7) in the Backlog milestone. Measured, not guessed: **12 findings** (11 × `set-state-in-effect`, 1 × `preserve-manual-memoization`) across 7 files; the other 12 compiler rules are already clean. It's scoped and ready to pull into a frontend milestone whenever one exists. Deployed state is unchanged — dev and prod are still on the `v3.4.1` tag. None of the above is live yet.

claude-bot commented

2026-07-17 03:52:33 +00:00

Author

Contributor

Correction to an earlier comment on this issue.

I flagged that prod's compose loads docker-compose.override.yml "(the dev port-bindings overlay)" and suggested confirming it was intended. That was a false alarm — please disregard it.

docker-compose.override.yml is intentionally empty:

# Intentionally empty.
#
# Quest Board now defaults to the production-safe compose stack in
# `docker-compose.yml`. Development behavior lives in `docker-compose.dev.yml`
# and must be opted into explicitly.

services: {}

Compose auto-loads it, which is why it shows up in com.docker.compose.project.config_files, but it contributes nothing. Both dev and prod correctly run the frontend-prod target from docker-compose.yml. Nothing to fix.

What is genuinely stale: the root CLAUDE.md still describes the file as docker-compose.override.yml Dev port bindings (auto-loaded), which is what led me astray. That description no longer matches the file and is worth correcting.

Separately, dev is now deployed to main (d339c41) so the v3.5.0 PWA work can be exercised. Verified through the real HTTPS origin (https://questboard-dev.rhoving.com): manifest serves as application/manifest+json, sw.js/registerSW.js/theme-init.js/icons all 200, the served index.html has zero inline scripts (so nothing trips the script-src 'self' CSP), and all 7 containers are healthy. Prod is untouched, still on v3.4.1.

**Correction to an earlier comment on this issue.** I flagged that prod's compose loads `docker-compose.override.yml` "(the dev port-bindings overlay)" and suggested confirming it was intended. **That was a false alarm — please disregard it.** `docker-compose.override.yml` is *intentionally empty*: ```yaml # Intentionally empty. # # Quest Board now defaults to the production-safe compose stack in # `docker-compose.yml`. Development behavior lives in `docker-compose.dev.yml` # and must be opted into explicitly. services: {} ``` Compose auto-loads it, which is why it shows up in `com.docker.compose.project.config_files`, but it contributes nothing. Both dev and prod correctly run the `frontend-prod` target from `docker-compose.yml`. Nothing to fix. **What is genuinely stale:** the root `CLAUDE.md` still describes the file as `docker-compose.override.yml Dev port bindings (auto-loaded)`, which is what led me astray. That description no longer matches the file and is worth correcting. --- Separately, **dev is now deployed to `main` (`d339c41`)** so the v3.5.0 PWA work can be exercised. Verified through the real HTTPS origin (`https://questboard-dev.rhoving.com`): manifest serves as `application/manifest+json`, `sw.js`/`registerSW.js`/`theme-init.js`/icons all 200, the served `index.html` has **zero** inline scripts (so nothing trips the `script-src 'self'` CSP), and all 7 containers are healthy. Prod is untouched, still on `v3.4.1`.

claude-bot commented

2026-07-17 15:40:03 +00:00

Author

Contributor

v3.5.0 tagged and deployed to both stacks.

Tag v3.5.0 (2ea6f26) — APP_VERSION/BOT_EXPECTED_APP_VERSION bumped to 3.5.0, changelog stamped. No API contract change (BOT_CONTRACT_VERSION stays 1). Forgejo release published with full notes.
dev and prod both on v3.5.0, reporting 3.5.0, /ready 200, alembic at head (no migration), all containers healthy. PWA verified through both public origins (questboard-dev.rhoving.com, questboard.rhoving.com): manifest served as application/manifest+json, 0 inline scripts, all artifacts 200.

CI note — registry images job: the tag's release.yml run failed in 25s at docker login. Root cause: release.yml needs two repo secrets and only REGISTRY_TOKEN was set — REGISTRY_USER was missing. Added REGISTRY_USER=rbrooks (the package owner, matching the token). The Forgejo release itself published fine (separate job), and the app is unaffected either way since both hosts build from source, not from published images.

The v3.5.0 images weren't built (secret added after the run). They'll build on the next tag automatically; to publish them now, re-run the failed job from the web UI (.../actions/runs/487). Not blocking anything.

**v3.5.0 tagged and deployed to both stacks.** - Tag `v3.5.0` (`2ea6f26`) — `APP_VERSION`/`BOT_EXPECTED_APP_VERSION` bumped to 3.5.0, changelog stamped. No API contract change (`BOT_CONTRACT_VERSION` stays 1). Forgejo release published with full notes. - **dev** and **prod** both on `v3.5.0`, reporting 3.5.0, `/ready` 200, alembic at head (no migration), all containers healthy. PWA verified through both public origins (`questboard-dev.rhoving.com`, `questboard.rhoving.com`): manifest served as `application/manifest+json`, 0 inline scripts, all artifacts 200. **CI note — registry `images` job:** the tag's `release.yml` run failed in 25s at `docker login`. Root cause: `release.yml` needs **two** repo secrets and only `REGISTRY_TOKEN` was set — `REGISTRY_USER` was missing. Added `REGISTRY_USER=rbrooks` (the package owner, matching the token). The Forgejo release itself published fine (separate job), and the app is unaffected either way since both hosts build from source, not from published images. The v3.5.0 images weren't built (secret added after the run). They'll build on the next tag automatically; to publish them now, re-run the failed job from the web UI (`.../actions/runs/487`). Not blocking anything.

claude-bot referenced this issue

2026-08-25 19:32:48 +00:00

[Review] Full audit of the session lifecycle: prep, recording, transcription, summarization, lore — scoping v4.0.0 #319

claude-bot added this to the v4.1.0 - Data Durability & Recovery milestone

2026-08-25 20:39:56 +00:00

claude-bot referenced this issue

2026-08-25 20:45:22 +00:00

[Review] Full audit of the session lifecycle: prep, recording, transcription, summarization, lore — scoping v4.0.0 #319

claude-bot commented

2026-09-01 17:03:10 +00:00

Author

Contributor

Two updates: item 4 is delegated to #433, and item 2's procedure has drifted from the code. Also removing this from v4.1.0 — see the end.

Item 4 (VAD trim) → #433

#433's synthetic-audio harness covers this better than a manual rehearsal would. It already measures VAD kept seconds as a first-class quantity (it has to — the quiet player's branch is decided by kept, not speech time), it runs against a real Whisper server, and it assembles the same scene two ways from one manifest. Trimmed-versus-untrimmed ordering, timestamps and speaker attribution fall out of that comparison deterministically and re-runnably, rather than as a one-off observation.

Nothing else here is covered by #433, and that is by design rather than oversight: it enters at POST /api/bot/sessions/{id}/audio with pre-rendered clips precisely so it never needs Discord. Items 1, 3 and 5 have nothing to do with audio, and item 2 is the one it structurally cannot cover — the whole point is the bot's RSS during live voice capture, and #433 never runs that path.

Item 2's expectations are stale

Written before #176, which moved the 48 kHz → 16 kHz mono conversion into the capture path. As written it looks for the wrong files and the wrong numbers:

item 2 says	actual, post-#176
`*_raw.wav` deleted at stop	raw tracks are `{audio_temp}/{session_id}_{user_id}_raw.16k-mono.s16le`, deleted as each is converted
"was ~0.6–0.7 GB/speaker-hour before"	roughly a sixth of that — the bot no longer holds Discord-format audio
RSS plateau is the thing being checked	still right, and still the point — #82's change is what this verifies

The RSS-plateau assertion is unchanged and remains the reason to run this: sample VmRSS from /proc/<bot_pid>/status every 30s across a 15–20 minute multi-speaker session and confirm it flattens rather than climbing per speaker-hour. Just expect the new filenames and the smaller on-disk figures.

Worth noting #176 also added recover_interrupted_recordings, so a bot killed mid-capture now converts and hands off its raw tracks at next startup. That is adjacent enough to item 2's setup to be worth exercising in the same session — kill the bot mid-recording, restart, confirm the session directory appears and the handoff posts.

Milestone

Removing this from v4.1.0 and leaving it unmilestoned. Live rehearsal needs a real Discord guild and a voice session, which is several days out, and every other issue in that milestone is done — there is no reason for this to hold the milestone open. It will be re-milestoned when it is scheduled rather than sitting as a permanent blocker.

Remaining scope after the above: four items — at-least-once bot delivery across restart (#81), disk-streamed recording memory with the corrected expectations (#82), healthchecks + AOF (#84), and the Redis pool auth round-trip (#96).

Two updates: **item 4 is delegated to #433**, and **item 2's procedure has drifted from the code**. Also removing this from v4.1.0 — see the end. ## Item 4 (VAD trim) → #433 #433's synthetic-audio harness covers this better than a manual rehearsal would. It already measures VAD `kept` seconds as a first-class quantity (it has to — the quiet player's branch is decided by `kept`, not speech time), it runs against a real Whisper server, and it assembles the same scene two ways from one manifest. Trimmed-versus-untrimmed ordering, timestamps and speaker attribution fall out of that comparison deterministically and re-runnably, rather than as a one-off observation. **Nothing else here is covered by #433**, and that is by design rather than oversight: it enters at `POST /api/bot/sessions/{id}/audio` with pre-rendered clips precisely so it never needs Discord. Items 1, 3 and 5 have nothing to do with audio, and item 2 is the one it structurally *cannot* cover — the whole point is the bot's RSS during live voice capture, and #433 never runs that path. ## Item 2's expectations are stale Written before #176, which moved the 48 kHz → 16 kHz mono conversion into the capture path. As written it looks for the wrong files and the wrong numbers: | item 2 says | actual, post-#176 | |---|---| | `*_raw.wav` deleted at stop | raw tracks are `{audio_temp}/{session_id}_{user_id}_raw.16k-mono.s16le`, deleted as each is converted | | "was ~0.6–0.7 GB/speaker-hour before" | roughly a sixth of that — the bot no longer holds Discord-format audio | | RSS plateau is the thing being checked | still right, and still the point — #82's change is what this verifies | The RSS-plateau assertion is unchanged and remains the reason to run this: sample `VmRSS` from `/proc/<bot_pid>/status` every 30s across a 15–20 minute multi-speaker session and confirm it flattens rather than climbing per speaker-hour. Just expect the new filenames and the smaller on-disk figures. Worth noting #176 also added `recover_interrupted_recordings`, so a bot killed mid-capture now converts and hands off its raw tracks at next startup. That is adjacent enough to item 2's setup to be worth exercising in the same session — kill the bot mid-recording, restart, confirm the session directory appears and the handoff posts. ## Milestone Removing this from v4.1.0 and leaving it unmilestoned. Live rehearsal needs a real Discord guild and a voice session, which is several days out, and every other issue in that milestone is done — there is no reason for this to hold the milestone open. It will be re-milestoned when it is scheduled rather than sitting as a permanent blocker. Remaining scope after the above: **four items** — at-least-once bot delivery across restart (#81), disk-streamed recording memory with the corrected expectations (#82), healthchecks + AOF (#84), and the Redis pool auth round-trip (#96).

claude-bot removed this from the v4.1.0 - Data Durability & Recovery milestone

2026-09-01 17:03:15 +00:00