[Ops] Live-stack rehearsal: bot delivery across restart, recording memory and crash recovery, healthchecks/AOF, Redis pool auth #1246

New issue

Open

opened 2026-10-27 17:38:35 +00:00 by claude-bot · 3 comments

claude-bot commented

2026-10-27 17:38:35 +00:00

Collaborator

Split from #175 (the v3.4.0 reliability checks), which is closed. These four checks still need a live stack and were never run. The VAD item went to the synthetic harness (#433).

At-least-once bot delivery across restart (#81). Stop the bot, publish an event (for example cancel a bot-connected session) and see XLEN qb:bot:events grow. Start the bot: exactly one post goes out and XPENDING is 0. Re-publish the same event_id: it is skipped. Kill the bot mid-handle and restart: the redelivered event is skipped. Recording live status still flows over recording:{guild_id}.
Disk-streamed recording memory (#82), with #176's expectations. Over a 15 to 20 minute multi-speaker session, sample VmRSS every 30 s and expect it to plateau. The raw tracks are {audio_temp}/{session_id}_{user_id}_raw.16k-mono.s16le, each deleted as it converts. In the same session, kill the bot mid-recording, restart it, and confirm recover_interrupted_recordings hands the session off and the channel is told (#393).
Healthchecks and AOF (#84). Follow the OPERATIONS.md checklist: pausing the worker flips it to unhealthy, and sessions survive docker compose restart redis with appendonly yes.
Redis pool auth round-trip (#96). Run login, then the session cookie, then authenticated requests, under concurrency, against a real Redis.

Where: items 3 and 4 can run on dev. Items 1 and 2 need a bot with a Discord token and a voice channel; the dev bot has none. That means prod in a quiet window, or a test guild. File any deviation as a bug.

Split from #175 (the v3.4.0 reliability checks), which is closed. These four checks still need a **live stack** and were never run. The VAD item went to the synthetic harness (#433). 1. **At-least-once bot delivery across restart (#81).** Stop the bot, publish an event (for example cancel a bot-connected session) and see `XLEN qb:bot:events` grow. Start the bot: **exactly one** post goes out and `XPENDING` is 0. Re-publish the same `event_id`: it is skipped. Kill the bot mid-handle and restart: the redelivered event is skipped. Recording live status still flows over `recording:{guild_id}`. 2. **Disk-streamed recording memory (#82), with #176's expectations.** Over a 15 to 20 minute multi-speaker session, sample `VmRSS` every 30 s and expect it to **plateau**. The raw tracks are `{audio_temp}/{session_id}_{user_id}_raw.16k-mono.s16le`, each deleted as it converts. In the same session, kill the bot mid-recording, restart it, and confirm `recover_interrupted_recordings` hands the session off and the channel is told (#393). 3. **Healthchecks and AOF (#84).** Follow the OPERATIONS.md checklist: pausing the worker flips it to unhealthy, and sessions survive `docker compose restart redis` with `appendonly yes`. 4. **Redis pool auth round-trip (#96).** Run login, then the session cookie, then authenticated requests, under concurrency, against a real Redis. **Where:** items 3 and 4 can run on dev. Items 1 and 2 need a bot with a Discord token and a voice channel; the dev bot has none. That means prod in a quiet window, or a test guild. File any deviation as a bug.

claude-bot added this to the v4.7.0 - Browse the Rules milestone

2026-10-27 17:38:35 +00:00

claude-bot referenced this issue

2026-10-27 17:38:50 +00:00

[Ops] Live-stack verification of v3.4.0 reliability changes #175

claude-bot commented

2026-11-03 15:32:34 +00:00

Author

Collaborator

Picking this up.

Items 3 and 4 (healthchecks and AOF; Redis pool auth under concurrency): run on a throwaway stack on docker-test built from main, with the production compose settings and the e2e seed, not on dev, which has real users. Results follow here.
Items 1 and 2 need the real bot, a voice channel and people talking for 15–20 minutes. How to run them is a question for the owner (Q218).

Picking this up. - **Items 3 and 4** (healthchecks and AOF; Redis pool auth under concurrency): run on a **throwaway stack** on docker-test built from main, with the production compose settings and the e2e seed, not on dev, which has real users. Results follow here. - **Items 1 and 2** need the real bot, a voice channel and people talking for 15–20 minutes. How to run them is a question for the owner (Q218).

claude-bot commented

2026-11-03 15:34:22 +00:00

Author

Collaborator

The owner's answer to Q218 (2026-11-03): (a). Items 1 and 2 will run on prod in a quiet window the owner picks, with the owner and one other person in voice for about 20 minutes. I drive the restarts. Timing depends on finding a second person, so it may be today or tomorrow.

This doesn't block 4.7.0. These checks test behaviour shipped in 3.4.0 (bot delivery, streamed recording, crash recovery), not anything new in 4.7.0. If the window comes after the release, items 1 and 2 run against 4.7.0 on prod, which is as good a test.

**The owner's answer to Q218 (2026-11-03): (a).** Items 1 and 2 will run on prod in a quiet window the owner picks, with the owner and one other person in voice for about 20 minutes. I drive the restarts. Timing depends on finding a second person, so it may be today or tomorrow. **This doesn't block 4.7.0.** These checks test behaviour shipped in 3.4.0 (bot delivery, streamed recording, crash recovery), not anything new in 4.7.0. If the window comes after the release, items 1 and 2 run against 4.7.0 on prod, which is as good a test.

claude-bot commented

2026-11-03 15:50:16 +00:00

Author

Collaborator

Items 3 and 4: results. These ran on a throwaway stack, ops1246 on docker-test, built from main at 3248a87a, with the base compose's redis, worker and healthchecks; the e2e overlay doesn't touch redis. quest-board was untouched and the stack is torn down.

Item 3, healthchecks and AOF (#84): PASS, with two checklist problems.

Worker healthcheck: after stopping the worker's main process (SIGSTOP), the probe failed three times with "No nodes replied within time constraint" and marked it unhealthy at +81 s. After SIGCONT it was healthy again at +21 s.
Checklist problem 1: docker compose exec worker kill -STOP 1 fails. The image has no kill binary.
Checklist problem 2: docker pause worker proves nothing. Docker 29 marks a paused container unhealthy by itself without running the probe.
AOF: appendonly yes, appendfsync everysec, the AOF files are in the named volume, and aof_last_write_status:ok.
Session survives a redis restart: a signed-in session still returned 200 after docker compose restart redis. Also tested beyond the checklist: after a SIGKILL of redis (no final snapshot), a session created just before still returned 200, so the AOF itself carries sessions.
Not done: "queued Celery tasks survive", because beat doesn't run in this stack.

Item 4, Redis pool auth (#96): PASS, with one finding.

Load: 1,000 authenticated requests from 6 sessions, 50 at a time, all returned 200 (p50 492 ms, p95 1.19 s). Of 600 /api/me checks, every one returned the right user. Redis received 44 connections for about 1,800 requests, with 0 rejected.
Restart mid-traffic: 2,269 requests; 2,251 returned 200 and 18 returned 500, all within the same third of a second as the restart. No 401s and no re-login: the same cookies worked straight after. The 500s are an uncaught redis.exceptions.ConnectionError in app/auth/session.py get_session. The body is still the standard {"detail": …} with no stack trace, but it reads as a server error rather than "try again".

The two findings are filed separately. Items 1 and 2 wait for the owner's voice window (Q218, answered (a)).

**Items 3 and 4: results.** These ran on a throwaway stack, `ops1246` on docker-test, built from main at 3248a87a, with the base compose's redis, worker and healthchecks; the e2e overlay doesn't touch redis. `quest-board` was untouched and the stack is torn down. **Item 3, healthchecks and AOF (#84): PASS, with two checklist problems.** - **Worker healthcheck:** after stopping the worker's main process (SIGSTOP), the probe failed three times with "No nodes replied within time constraint" and marked it unhealthy at +81 s. After SIGCONT it was healthy again at +21 s. - **Checklist problem 1:** `docker compose exec worker kill -STOP 1` fails. The image has no `kill` binary. - **Checklist problem 2:** `docker pause worker` proves nothing. Docker 29 marks a paused container unhealthy by itself without running the probe. - **AOF:** `appendonly yes`, `appendfsync everysec`, the AOF files are in the named volume, and `aof_last_write_status:ok`. - **Session survives a redis restart:** a signed-in session still returned 200 after `docker compose restart redis`. Also tested beyond the checklist: after a SIGKILL of redis (no final snapshot), a session created just before still returned 200, so the AOF itself carries sessions. - **Not done:** "queued Celery tasks survive", because beat doesn't run in this stack. **Item 4, Redis pool auth (#96): PASS, with one finding.** - **Load:** 1,000 authenticated requests from 6 sessions, 50 at a time, all returned 200 (p50 492 ms, p95 1.19 s). Of 600 `/api/me` checks, every one returned the right user. Redis received 44 connections for about 1,800 requests, with 0 rejected. - **Restart mid-traffic:** 2,269 requests; 2,251 returned 200 and **18 returned 500**, all within the same third of a second as the restart. No 401s and no re-login: the same cookies worked straight after. The 500s are an uncaught `redis.exceptions.ConnectionError` in `app/auth/session.py` `get_session`. The body is still the standard `{"detail": …}` with no stack trace, but it reads as a server error rather than "try again". The two findings are filed separately. Items 1 and 2 wait for the owner's voice window (Q218, answered (a)).

claude-bot referenced this issue

2026-11-03 15:50:21 +00:00

A Redis blip answers signed-in requests with 500, not 503; and the OPERATIONS.md worker-healthcheck step doesn't work #1326