[Ops] Live-stack verification of v3.4.0 reliability changes #175
Closed
opened 2026-07-16 02:14:35 +00:00 by claude-bot
·
6 comments
No Branch/Tag specified
main
feat/1328-fill-it-in
feat/1328-fill-frontend
feat/1328-fill-backend
feat/9-foundry-push
fix/9-foundry-live
feat/9-foundry-push-frontend
renovate/celery-5.x
feat/5-foundry-connect
feat/9-foundry-push-backend
feat/9-foundry-push-module
feat/5-foundry-frontend
feat/5-foundry-backend
feat/5-foundry-module
release/v4.8.0-rc1
docs/d341-convert-draft
feat/958-update-to-version
feat/958-convert-sheet
feat/958-pathbuilder-import-frontend
fix/1360-heading-order
feat/958-pathbuilder-import-backend
feat/11-weekly-digest
feat/1341-auto-update
docs/d332-superseded
chore/1345-names-cleanup
feat/1342-mentioned-pages
feat/1328-recipes
fix/1346-pgdg-key
perf/1247-shared-recording-stream
chore/132-gitleaks-allowlist
release/v4.7.1
spike/foundry-module
release/v4.7.1-rc1
docs/decisions-d332-d337
fix/bot-certifi
chore/release-4.7.0
chore/release-4.7.0-rc4
chore/deps-4.7.0
chore/release-4.7.0-rc3
fix/1280-quote-must-say-it
fix/1257-names-row-squeeze
chore/release-4.7.0-rc2
fix/1245-pg-upgrade-count
chore/release-4.7.0-rc1
feat/1257-names-in-prep
feat/1257-be
feat/1257-fe
feat/1257-rating-fixes
design/1328-fill-it-in
feat/1257-name-splits
fix/1326-redis-503
feat/1280-fact-suggestions
feat/1280-be
feat/1280-fe
design/1280-fact-suggestions
feat/1279-infobox
feat/1279-be
feat/1279-fe
feat/1321-shelf-copies
design/1279-infobox
feat/1243-table-find
feat/1257-who-rated
feat/1243-prep-drawer
feat/1243-sheet-pickers
design/1243-browse-in-place
feat/1315-hotwords-default
feat/1242-system-filters
feat/1257-more-name-styles
feat/506-leg3-hotwords
design/1242-system-filters
fix/1302-packs-polish
feat/1241-rules-library
design/1241-rules-library
feat/1245-postgres-18
integ/4.6.2
fix/1263-map-ui-2
fix/1263-map-server
fix/1263-map-ui-1
fix/1261-editor-link-menu
fix/1229-axe-4.14-name-mismatch
renovate/axllent-mailpit-1.x
renovate/eslint-monorepo
renovate/authlib-1.x
chore/release-4.6.0
chore/release-4.6.0-rc2
fix/460-rc2-screens
fix/460-rc2-server
fix/release-prerelease-exit
chore/release-4.6.0-rc1
feat/125-campaign-templates
feat/126-calendar
feat/1181-pictures-and-files
feat/1190-event-people
integ/4.6.0
feat/1191-line-of-the-night
feat/1200-starter-packs
chore/bot-multidict-and-playwright-group
fix/1202-this-week-answered-count
fix/122-124-leftovers
fix/1178-get-routes-that-write
fix/1203-journeys-check-names
design/125-campaign-templates
feat/126-calendar-engine
design/126-in-world-calendar
fix/1189-cited-stamp-in-range
fix/1189-v4-looser-check
fix/1189-richer-beats
fix/1113-export-bind-lists
fix/1154-release-notes-by-file
chore/release-4.5.0
chore/release-4.5.0-rc12
fix/1135-wiki-markup-as-text
fix/demo-usage-cost-in-window
chore/release-4.5.0-rc11
fix/1121-sheet-consistency
fix/1121-sheet-5e-terms-and-card-hp
copy/1122-busiest-pages
fix/1121-relevant-choices
fix/1121-wiki-create-npc-save-sheet-links
fix/1029-menu-stays-on-screen
docs/prototype-transcript-bin-wording
fix/1023-editor-save-row-back-scroll
fix/1020-search-exact-first
fix/1020-table-prep
fix/1020-shell-wordmark-footer-bell
fix/1020-wiki-article-party
fix/872-admin-kept-until-phone-tables
feat/l13-milestone-level-waiting
fix/1000-1001-bell-older-setup-context
fix/999-pronouns-in-prompts
fix/l19-other-class-feats
fix/1010-transcript-window-ordinary-confirm
docs/d254-d269-decisions
fix/702-member-search
fix/680-transcript-bin
fix/993-unidentified-prompts
fix/841-redis-timeouts
fix/990-binned-not-exported
fix/830-export-prep-shelf
fix/deps-audit-2026-10-01
fix/877-title-reads
fix/798-audio-received-at
build/lane11-17-j7
build/lane11-16-levelup
build/lane11-14-seams
build/lane11-14-seed
build/lane11-15-start
build/lane11-13-old-block-out
fix/980-review-r3
build/lane11-12-block
build/lane11-8-packs
fix/demo-seed-canonical-names
build/lane11-11-hot-half
docs/l11-decisions
build/lane11-7-convert
build/lane11-10-editing
fix/npm-audit-brace-expansion
build/lane11-9-sheet-page
fix/bot-pyjwt
build/lane11-6-content
fix/lane11-rank-names
build/lane11-2b-language
build/lane11-4-js-evaluator
build/lane11-3-server
build/lane11-5a-pf2e
build/lane11-5b-5e
feat/design-admin
build/lane11-2-evaluator
build/lane11-1-sheet-pieces
build/lane11-0-drawing
fix/943-quest-objectives-fk
fix/prepsheet-shelf-flake
fix/945-leftovers
build/demo-gm-copy
fix/901-email-landings
build/lane10-8-journeys
build/demo-seed-lanes-8-10
fix/887-gm-own-confirmation
build/lane9-5-j1-admin
fix/918-audience-click
fix/useasync-overtaken
ci/939-critical-path
test/937-backend-suite-speed
chore/claude-orchestration
fix/924-xp-gm-only
fix/session-stale-poll
docs/decisions-d239-d242
fix/930-pypdf
build/lane8-journeys
build/925-quest-editor
fix/894-private-page-pictures
build/lane8-p4-drafts
build/lane8-p7-map
build/lane8-p6-handouts-rules
fix/wiki-fixture-names
build/lane10-d227-test-usage
fix/rsvp-horizon-test-tuesday
fix/media-volume-owner
build/lane8-p3-editor
fix/912-unidentified-rename
build/lane8-p5-party-quests
build/lane8-handout-takeback
build/lane10-pickers
build/lane8-p2-article
build/lane8-6-map
build/lane10-7b-email-pages
build/lane8-5-handouts
build/lane10-7a-test-the-bot-pages
build/lane8-p1-pages
build/lane8-4-quests
build/lane8-2b-feeds
build/lane8-3-parties
build/lane8-2a-links
build/lane10-5-commands
build/lane9-6-seed
build/lane10-4b-reactions
build/lane9-4d-pages
build/lane9-4c-pages
build/lane10-4a-posts
build/lane10-3-email
fix/891-web-fonts
build/lane9-4b-pages
build/lane10-6-test-the-bot
build/lane10-2-notify-out
build/lane8-1-wiki-text
build/lane9-4a-frame
design/lane10-drawings
fix/j6-budget
fix/j5-phone-flake
fix/869-test-leaks
fix/843-record-start-followups
build/lane7-all
build/lane6-all
build/lane9-b2-server
chore/changelog-fragments
docs/decisions-d218
build/lane9-a-owner-overview
fix/rule-card-level-label
fix/834-retention-semantics
fix/o3c-record-without-announcement
docs/decisions-d179
docs/decisions-d131-d170
fix/851-commit-before-response
build/lane6-j3-seed
build/lane9-b1-d67-identity
fix/841-blocking-loop
build/lane9-c-backups-audit
fix/vote-race-and-start-commit
fix/recording-announcement
fix/proposal-read-leak
fix/last-admin-guard
build/lane6-p4c-planning
build/lane6-p4b-margin
build/lane6-p4a-sheet
build/lane5-j4-seed
build/lane6-base
build/lane7-pages-talk
build/lane5-pages
build/lane7-pages-safety
build/lane7-p3-safety
build/lane7-p2-notify
build/lane6-p3-planning
build/lane6-p2-history
build/lane6-p1b-items
build/lane7-p1-threads
build/lane5-p6-faces
build/lane6-p1a-sheet
build/lane5-p5-find
build/lane5-p4a-board
build/lane5-p4b-scenes
build/lane5-p3-live
build/lane5-base
build/lane5-p2-cards
docs/lane5-decisions
build/review-r2-wide
fix/review-r1
build/lane5-p1-scenes
build/demo-seed
build/lane4-journeys-j5
build/lane4-page-review-b
build/lane4-page-review-a
build/lane4-page-read
build/lane4-journeys
build/lane4-page-sess2
build/lane4-page-sess1
build/lane4-notes
build/lane4-discord-on-approval
build/lane4-scheduling
build/lane4-versions
build/760-session-number
build/lane4-decisions
build/797-erase-no-regenerate
build/test-race-audit
build/lane3-completions
build/794-dark-only
build/796-titles-scroll
build/791-retention-anchor
build/a11y-sweep
build/shell-followups
build/retention-preview
build/753-backup-readback
docs/d91-d93
build/lane3-j1
build/ci-harness-paths-selftest
fix/useauth-test-race
build/ci-harness-paths
build/777-ai-settings-read
build/lane3-settings-c
build/lane3-settings-b
build/lane3-j6
build/lane3-sessionform
build/lane3-settings-a
build/lane3-members
build/lane3-thisweek
build/lane3-story
chore/87-align-python-3.12
v4.8.0-rc1
v4.7.1
v4.7.1-rc1
v4.7.0
v4.7.0-rc4
v4.7.0-rc3
v4.7.0-rc2
v4.7.0-rc1
v4.6.2
v4.6.2-rc4
v4.6.2-rc3
v4.6.2-rc2
v4.6.2-rc1
v4.6.1
v4.6.1-rc1
v4.6.0
v4.6.0-rc2
v4.6.0-rc1
v4.5.3
v4.5.3-rc1
v4.5.2
v4.5.2-rc1
v4.5.1
v4.5.1-rc2
v4.5.1-rc1
v4.5.0
v4.5.0-rc12
v4.5.0-rc11
v4.5.0-rc10
v4.5.0-rc9
v4.5.0-rc8
v4.5.0-rc7
v4.5.0-rc6
v4.5.0-rc5
v4.5.0-rc4
v4.5.0-rc3
v4.5.0-rc2
v4.5.0-rc1
v4.4.1
v4.4.0
v4.3.0
v4.2.3
v4.2.2
v4.2.1
v4.2.0
v4.1.2
v4.1.1
v4.1.0
v4.0.1
v4.0.0
v3.11.4
v3.11.3
v3.11.2
v3.11.1
v3.11.0
v3.10.0
v3.9.1
v3.9.0
v3.8.0
v3.7.0
v3.6.1
v3.6.0
v3.5.0
v3.4.1
v3.4.0
v3.3.0
v3.2.0
v3.1.0
v3.0.0
v2.1.0
v2.0.0
v1.1.0
v1.0.0
v0.12.0
v0.11.0
v0.10.0
v0.9.1
v0.3.3
v0.9.0
v0.8.0
v0.3.2
v0.3.1
v0.3.0
v0.7.0
v0.6.0
v0.5.0
v0.4.0
v0.2.1
v0.2.0
v0.1.1
v0.1.0
Labels
Clear labels
backend
Issue related to the backend
bot
Issue related to the bot
bug
Something is not working
duplicate
This issue or pull request already exists
enhancement
New feature
frontend
Issue related to the frontend
help wanted
Need some help
invalid
Something is wrong
question
More information is needed
recording
Issue related to the recording system
security
Security hardening / vulnerability work
wontfix
This won't be fixed
No labels
backend
bot
bug
duplicate
enhancement
frontend
help wanted
invalid
question
recording
security
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
rbrooks/Quest-Board#175
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
Several v3.4.0 reliability changes have behaviours that cannot be exercised in the mock/unit test harness (no live Redis, no real Discord voice, no multi-container runtime). Each was shipped with a documented manual procedure; this issue tracks actually rehearsing them once on a live/staging stack before treating v3.4.0 as production-verified.
Checklist
1. At-least-once bot delivery — cross-restart (#81)
Follow the procedure in the PR #166 comment:
docker compose stop bot.redis-cli -n 0 XLEN qb:bot:eventsincrements and no Discord post yet.docker compose start bot→ exactly one embed posts (backlog drain);XPENDING qb:bot:events qb-bot-workers→ 0.event_id→ skipped (debug log), no second post;TTL qb:bot:handled:<id>≈ 3600.recording:{guild_id}pub/sub.2. Disk-streamed recording — memory + e2e (#82)
VmRSSfrom/proc/<bot_pid>/statusevery 30s.audio_tempgrow instead./record stop→ no RSS spike (oldgetvalue()copy is gone);{session_dir}/{user_id}.wavappear,*_raw.wavdeleted.ffprobea converted WAV = 16 kHz mono s16 (unchanged format).3. Healthchecks + AOF (#84)
Per the "Verifying Healthchecks and AOF Persistence" checklist in
docs/OPERATIONS.md:docker compose psshows health for worker and beat;docker pause(or SIGSTOP) the worker → flips unhealthy within a few intervals.docker compose restart rediswith AOF; confirmappendonly yesand AOF files on disk.4. VAD trim (#101)
vad_trim_enabledin Admin → Bot Settings; process a recorded session."VAD trimmed Xh -> Yh"log line, and that the merged transcript ordering/timestamps/speaker-attribution match an untrimmed run of the same audio.5. Redis pool auth round-trip (#96)
Acceptance
v3.4.1 cut — healthcheck deviation from item 3 (#84) fixed & deployed
Recording a deviation found while rehearsing item 3 (Healthchecks + AOF, #84) on the live stack, plus the patch release that resolved it.
Deviation (bug): the #84 frontend healthcheck used
wget --spider http://localhost/. Thenginx:alpineimage's/etc/hostsmapslocalhostto IPv6::1only, but nginx binds IPv40.0.0.0:80, so the probe got connection-refused and the frontend container flapped unhealthy even though nginx was serving fine. (backend/bot use Debian images wherelocalhost→127.0.0.1, so they were unaffected.)Fix: probe
http://127.0.0.1/instead — commit1051f19(fix(compose): frontend healthcheck probes 127.0.0.1 not localhost).Released as v3.4.1 (
b5b53ff): bumpedAPP_VERSION+BOT_EXPECTED_APP_VERSIONto3.4.1and recorded the fix in the changelog. No API contract change (BOT_CONTRACT_VERSIONstays 1). Forgejo release published; theimagesjob built/pushed the:3.4.1images.Deployed (build-from-source,
git pull --ff-only→v3.4.1→docker compose up -d --build):docker-test, 10.1.1.14)docker-host, 10.1.1.7)v3.4.1(b5b53ff)v3.4.1(b5b53ff)APP_VERSIONBOT_EXPECTED/readye5f6g7h8i9j0(head, no new migration)Item 3 healthcheck sub-check is now verified green on both stacks (worker/beat/frontend health all reporting correctly). The AOF-persistence half of item 3, and items 1/2/4/5, remain to be rehearsed — leaving this issue open.
Two observations to follow up separately (not filed yet):
docker-compose.override.yml(the dev port-bindings overlay) as one of itsproject.config_files— pre-existing, not from this deploy, but worth confirming it's intended.Unrelated to this issue's checklist, but noting it here since this is the live-verification thread and the v3.5.0 frontend work has now landed on
main(d339c41, 11 commits pastv3.4.1).v3.5.0 — Frontend Platform & PWA is code-complete (milestone: 0 open / 4 closed):
/apiNetworkOnlyFrontend suite went 69 → 106 tests. Post-merge CI green on
mainfor each.Two things that need a human before a v3.5.0 release:
application/manifest+json, 192/512any+maskableicons, SW registration, 0 CSP-blocked inline scripts — all confirmed against a realfrontend-prodimage) rather than by running Lighthouse itself. Offline behaviour likewise comes from the generated precache manifest and strategies, not from toggling a real browser offline.webapp/frontend/public/currently holds generated QB-monogram art. Drop real art in with the same filenames — no code change needed.Also filed #182 (adopt the React Compiler lint rules from eslint-plugin-react-hooks v7) in the Backlog milestone. Measured, not guessed: 12 findings (11 ×
set-state-in-effect, 1 ×preserve-manual-memoization) across 7 files; the other 12 compiler rules are already clean. It's scoped and ready to pull into a frontend milestone whenever one exists.Deployed state is unchanged — dev and prod are still on the
v3.4.1tag. None of the above is live yet.Correction to an earlier comment on this issue.
I flagged that prod's compose loads
docker-compose.override.yml"(the dev port-bindings overlay)" and suggested confirming it was intended. That was a false alarm — please disregard it.docker-compose.override.ymlis intentionally empty:Compose auto-loads it, which is why it shows up in
com.docker.compose.project.config_files, but it contributes nothing. Both dev and prod correctly run thefrontend-prodtarget fromdocker-compose.yml. Nothing to fix.What is genuinely stale: the root
CLAUDE.mdstill describes the file asdocker-compose.override.yml Dev port bindings (auto-loaded), which is what led me astray. That description no longer matches the file and is worth correcting.Separately, dev is now deployed to
main(d339c41) so the v3.5.0 PWA work can be exercised. Verified through the real HTTPS origin (https://questboard-dev.rhoving.com): manifest serves asapplication/manifest+json,sw.js/registerSW.js/theme-init.js/icons all 200, the servedindex.htmlhas zero inline scripts (so nothing trips thescript-src 'self'CSP), and all 7 containers are healthy. Prod is untouched, still onv3.4.1.v3.5.0 tagged and deployed to both stacks.
v3.5.0(2ea6f26) —APP_VERSION/BOT_EXPECTED_APP_VERSIONbumped to 3.5.0, changelog stamped. No API contract change (BOT_CONTRACT_VERSIONstays 1). Forgejo release published with full notes.v3.5.0, reporting 3.5.0,/ready200, alembic at head (no migration), all containers healthy. PWA verified through both public origins (questboard-dev.rhoving.com,questboard.rhoving.com): manifest served asapplication/manifest+json, 0 inline scripts, all artifacts 200.CI note — registry
imagesjob: the tag'srelease.ymlrun failed in 25s atdocker login. Root cause:release.ymlneeds two repo secrets and onlyREGISTRY_TOKENwas set —REGISTRY_USERwas missing. AddedREGISTRY_USER=rbrooks(the package owner, matching the token). The Forgejo release itself published fine (separate job), and the app is unaffected either way since both hosts build from source, not from published images.The v3.5.0 images weren't built (secret added after the run). They'll build on the next tag automatically; to publish them now, re-run the failed job from the web UI (
.../actions/runs/487). Not blocking anything.Two updates: item 4 is delegated to #433, and item 2's procedure has drifted from the code. Also removing this from v4.1.0 — see the end.
Item 4 (VAD trim) → #433
#433's synthetic-audio harness covers this better than a manual rehearsal would. It already measures VAD
keptseconds as a first-class quantity (it has to — the quiet player's branch is decided bykept, not speech time), it runs against a real Whisper server, and it assembles the same scene two ways from one manifest. Trimmed-versus-untrimmed ordering, timestamps and speaker attribution fall out of that comparison deterministically and re-runnably, rather than as a one-off observation.Nothing else here is covered by #433, and that is by design rather than oversight: it enters at
POST /api/bot/sessions/{id}/audiowith pre-rendered clips precisely so it never needs Discord. Items 1, 3 and 5 have nothing to do with audio, and item 2 is the one it structurally cannot cover — the whole point is the bot's RSS during live voice capture, and #433 never runs that path.Item 2's expectations are stale
Written before #176, which moved the 48 kHz → 16 kHz mono conversion into the capture path. As written it looks for the wrong files and the wrong numbers:
*_raw.wavdeleted at stop{audio_temp}/{session_id}_{user_id}_raw.16k-mono.s16le, deleted as each is convertedThe RSS-plateau assertion is unchanged and remains the reason to run this: sample
VmRSSfrom/proc/<bot_pid>/statusevery 30s across a 15–20 minute multi-speaker session and confirm it flattens rather than climbing per speaker-hour. Just expect the new filenames and the smaller on-disk figures.Worth noting #176 also added
recover_interrupted_recordings, so a bot killed mid-capture now converts and hands off its raw tracks at next startup. That is adjacent enough to item 2's setup to be worth exercising in the same session — kill the bot mid-recording, restart, confirm the session directory appears and the handoff posts.Milestone
Removing this from v4.1.0 and leaving it unmilestoned. Live rehearsal needs a real Discord guild and a voice session, which is several days out, and every other issue in that milestone is done — there is no reason for this to hold the milestone open. It will be re-milestoned when it is scheduled rather than sitting as a permanent blocker.
Remaining scope after the above: four items — at-least-once bot delivery across restart (#81), disk-streamed recording memory with the corrected expectations (#82), healthchecks + AOF (#84), and the Redis pool auth round-trip (#96).
Closing. Item 4 (VAD) is covered by the synthetic harness (#433). The four live-stack checks still to run are split into #1246 in v4.7.0, with item 2's expectations corrected for #176.