v4.5.0: upgrade rehearsal on a copy of prod, final cut and prod deploy #1140

New issue

Closed

opened 2026-10-12 22:38:04 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-10-12 22:38:04 +00:00

Collaborator

The release steps for v4.5.0, in the owner's order (10 October; Q164 and Q165): review fixes → prereleases to dev → all lanes signed (done, 12 October) → upgrade rehearsal on a copy of prod's database → cut 4.5.0 → prod, only on the owner's say.

1. Upgrade rehearsal: done, 2026-10-12 about 22:30 to 22:40 UTC. It passed.

The owner approved the read on prod and the rehearsal in chat on 12 October.

Prod before (read-only): app 4.4.1, PostgreSQL 16.15, database 17 MB, Alembic a0f1a2b3c4d5, 56 tables (54 owned by questboard_migrate, 2 by questboard_admin), 7 users, 2 campaigns, 22 sessions, 56 wiki pages. Nothing on prod was written or restarted; one pg_dump -Fc was read from its database container and streamed straight to the dev server.

The copy: a separate compose project qb-rehearsal on the dev server, from a git archive of v4.5.0-rc12, with its own made-up secrets, its own database and Redis, no published ports, and only db, redis and (after the migration) backend ever started: no worker, beat, bot or frontend, so nothing could send a reminder, an email or a Discord message from prod's data.

What was run: scripts/migrate-safe.sh (what make migrate-safe runs), from the rc12 images:

db_preflight: passed (1 Alembic revision, 7 users, 2 campaigns).
backup_now: wrote a dump.
alembic upgrade head: 58 revisions, a0f1a2b3c4d5 → d1054a2b3c4e, no errors. The whole script took 33 seconds. Backfills reported: parties 2, members 5, loot 4, custody 5; quests as pages 2; NPC tracker fold 0 pages; decided-in 0 posts.

After: 112 tables; users 7, campaigns 2, sessions 22 (none without a number), wiki pages 58 (56 + the 2 quests that became pages), parties 2, character sheets 1 (a converted stat block), 6 sheet definitions, both campaigns have an owner. The rc12 backend started on the migrated copy, /ready answered {"status":"ok"}, /api/meta 4.5.0-rc12, alembic heads = the database's revision. Its log's only errors were the expected one: three encrypted settings (whisper_config, llm_config, bot_api_key) could not be decrypted, because the rehearsal used its own SECRET_KEY, not prod's. On prod the key does not change, so this does not apply there.

Cleaned up: docker compose down -v for qb-rehearsal, the dump, the backup the script wrote and the work directory all deleted; no container, volume or network of that project is left. The dev stack itself was not touched (still rc12, healthy).

Two things the rehearsal showed that matter for the real upgrade

Step 2 of migrate-safe failed on the first try because ./backups did not exist and Docker created it root-owned, which the app's user cannot write. On prod that folder already exists and is the app user's (make backup-now writes there), so this should not occur, but check ls -ld backups before starting; the migrate service repairs it as root, but it runs after the backup step.
Not rehearsed: the bot, the worker and beat against the upgraded data, real sign-in, and anything using the three encrypted settings. BOT_CONTRACT_VERSION goes 1 → 2, so the bot and backend images must be updated together (the tag's compose file does that).

2. Final cut: not started

make changelog, the ## [4.5.0] — date header with its preamble (58 migrations since 4.4.1; BOT_CONTRACT_VERSION 1 → 2), versions to 4.5.0, release PR, tag, images (a final also moves :latest).

3. Prod deploy: only on the owner's say

make backup-now, git fetch --tags && git checkout v4.5.0, make migrate-safe, docker compose up -d, verify /api/meta, then the lane 10 Discord and email checks, and a count of wiki pages saved by the old Instant NPC path with a readable secret (a line starting "Hides: ").

The release steps for v4.5.0, in the owner's order (10 October; Q164 and Q165): review fixes → prereleases to dev → all lanes signed (done, 12 October) → **upgrade rehearsal on a copy of prod's database** → cut 4.5.0 → prod, only on the owner's say. ## 1. Upgrade rehearsal: done, 2026-10-12 about 22:30 to 22:40 UTC. It passed. The owner approved the read on prod and the rehearsal in chat on 12 October. **Prod before (read-only):** app 4.4.1, PostgreSQL 16.15, database 17 MB, Alembic `a0f1a2b3c4d5`, 56 tables (54 owned by `questboard_migrate`, 2 by `questboard_admin`), 7 users, 2 campaigns, 22 sessions, 56 wiki pages. Nothing on prod was written or restarted; one `pg_dump -Fc` was read from its database container and streamed straight to the dev server. **The copy:** a separate compose project `qb-rehearsal` on the dev server, from a `git archive` of `v4.5.0-rc12`, with its own made-up secrets, its own database and Redis, no published ports, and **only `db`, `redis` and (after the migration) `backend` ever started**: no worker, beat, bot or frontend, so nothing could send a reminder, an email or a Discord message from prod's data. **What was run:** `scripts/migrate-safe.sh` (what `make migrate-safe` runs), from the rc12 images: 1. `db_preflight`: passed (1 Alembic revision, 7 users, 2 campaigns). 2. `backup_now`: wrote a dump. 3. `alembic upgrade head`: **58 revisions, `a0f1a2b3c4d5` → `d1054a2b3c4e`, no errors.** The whole script took 33 seconds. Backfills reported: parties 2, members 5, loot 4, custody 5; quests as pages 2; NPC tracker fold 0 pages; decided-in 0 posts. **After:** 112 tables; users 7, campaigns 2, sessions 22 (none without a number), wiki pages 58 (56 + the 2 quests that became pages), parties 2, character sheets 1 (a converted stat block), 6 sheet definitions, both campaigns have an owner. The rc12 backend started on the migrated copy, `/ready` answered `{"status":"ok"}`, `/api/meta` `4.5.0-rc12`, `alembic heads` = the database's revision. Its log's only errors were the expected one: three encrypted settings (`whisper_config`, `llm_config`, `bot_api_key`) could not be decrypted, because the rehearsal used its own `SECRET_KEY`, not prod's. On prod the key does not change, so this does not apply there. **Cleaned up:** `docker compose down -v` for `qb-rehearsal`, the dump, the backup the script wrote and the work directory all deleted; no container, volume or network of that project is left. The dev stack itself was not touched (still rc12, healthy). **Two things the rehearsal showed that matter for the real upgrade** - Step 2 of `migrate-safe` failed on the first try because `./backups` did not exist and Docker created it root-owned, which the app's user cannot write. On prod that folder already exists and is the app user's (`make backup-now` writes there), so this should not occur, but check `ls -ld backups` before starting; the `migrate` service repairs it as root, but it runs after the backup step. - Not rehearsed: the bot, the worker and beat against the upgraded data, real sign-in, and anything using the three encrypted settings. `BOT_CONTRACT_VERSION` goes 1 → 2, so the bot and backend images must be updated together (the tag's compose file does that). ## 2. Final cut: not started `make changelog`, the `## [4.5.0] — date` header with its preamble (58 migrations since 4.4.1; `BOT_CONTRACT_VERSION` 1 → 2), versions to `4.5.0`, release PR, tag, images (a final also moves `:latest`). ## 3. Prod deploy: only on the owner's say `make backup-now`, `git fetch --tags && git checkout v4.5.0`, `make migrate-safe`, `docker compose up -d`, verify `/api/meta`, then the lane 10 Discord and email checks, and a count of wiki pages saved by the old Instant NPC path with a readable secret (a line starting "Hides: ").

claude-bot referenced this issue

2026-10-12 22:48:38 +00:00

chore(release): v4.5.0 #1141

claude-bot referenced this issue

2026-10-12 23:26:15 +00:00

[Hosted] Operations for a paid service #1152

claude-bot commented

2026-10-12 23:50:54 +00:00

Author

Collaborator

2. Final cut: done, 2026-10-12 23:38 UTC

PR #1141 (changelog gathered: 188 fragments; ## [4.5.0] — 2026-10-12 with the upgrade notes; versions to 4.5.0) passed all CI and merged (cfd25e77); v4.5.0 tagged; the three images published at :4.5.0 and :latest.

The workflow's "Create Forgejo release" job failed ("argument list too long": the section is 344,457 characters), so the release page was created by hand through the API with the upgrade notes and a link to the changelog: https://git.rhoving.com/rbrooks/Quest-Board/releases/tag/v4.5.0 . The workflow fix is #1154.

3. Prod deploy: done, 2026-10-12 23:44 to 23:50 UTC, on the owner's say ("Yes, let's cut that and deploy it.")

On the production host, in /home/ryan/Quest-Board:

make backup-now → backups/questboard_manual_20261012T234438Z.dump.
The checkout had an untracked docker-compose.images.yml (4.4.1's tree did not carry the file), which would have blocked the checkout; copied to /home/ryan/quest-board-images-4.4.1.yml.bak and renamed in place to docker-compose.images.yml.4.4.1. git fetch --tags && git checkout v4.5.0.
make migrate-safe from the 4.5.0 images: preflight passed (Alembic a0f1a2b3c4d5, 7 users, 2 campaigns); a second backup questboard_manual_20261012T234701Z.dump; 58 revisions to d1054a2b3c4e, the same backfill numbers as the rehearsal (parties 2, members 5, loot 4, custody 5; quests as pages 2). 133 seconds including the image pull. The 4.4.1 services kept running throughout.
docker compose pull && docker compose up -d: all seven services healthy on the 4.5.0 images within about a minute.

Verified

/ready → {"status":"ok"}; /api/meta → {"version":"4.5.0"}, from inside the stack and at https://questboard.rhoving.com/api/meta .
The bot: "Version handshake OK: backend 4.5.0, bot API contract v2", connected to Discord and logged in.
No error, traceback, refusal or 5xx in the backend, worker, beat, bot or frontend logs in the first minutes; no "cannot be decrypted" line (prod's SECRET_KEY is unchanged).
Counts: 7 users, 2 campaigns, 22 sessions (none without a number), 58 wiki pages (56 + the 2 quests), 2 parties, 6 sheet definitions, both campaigns have an owner.

Not verified by me

Nothing was looked at in a browser on prod, and nobody has signed in since the upgrade.
No recording, transcription, summary, reminder, email or Discord post has run on 4.5.0 yet.
Settings named in 4.5.0's .env.example that prod's .env does not set, so their defaults apply: BOT_EVENTS_MAXLEN, ENVIRONMENT, MEDIA_ROOT, MEDIA_URL, POSTGRES_APP_USER, RCLONE_CONFIG_PATH, VITE_ALLOWED_HOSTS. The app started and passed its own configuration checks; worth a read by the owner, ENVIRONMENT especially.

The owner's, on prod: lane 10's Discord and email pieces (posts and reactions, DMs and slash commands, the emails, stop-these-emails, answering from an email, the calendar invite), a sign-in, and the starter content packs for character sheets.

If it had to be undone: the two dumps above are from before the migrations; going back means restoring one, checking out v4.4.1, putting docker-compose.images.yml.4.4.1 back as docker-compose.images.yml, and docker compose up -d. Anything written after the upgrade would be lost.

## 2. Final cut: done, 2026-10-12 23:38 UTC PR #1141 (changelog gathered: 188 fragments; `## [4.5.0] — 2026-10-12` with the upgrade notes; versions to 4.5.0) passed all CI and merged (cfd25e77); `v4.5.0` tagged; the three images published at `:4.5.0` and `:latest`. The workflow's "Create Forgejo release" job failed ("argument list too long": the section is 344,457 characters), so the release page was created by hand through the API with the upgrade notes and a link to the changelog: https://git.rhoving.com/rbrooks/Quest-Board/releases/tag/v4.5.0 . The workflow fix is #1154. ## 3. Prod deploy: done, 2026-10-12 23:44 to 23:50 UTC, on the owner's say ("Yes, let's cut that and deploy it.") On the production host, in `/home/ryan/Quest-Board`: 1. `make backup-now` → `backups/questboard_manual_20261012T234438Z.dump`. 2. The checkout had an untracked `docker-compose.images.yml` (4.4.1's tree did not carry the file), which would have blocked the checkout; copied to `/home/ryan/quest-board-images-4.4.1.yml.bak` and renamed in place to `docker-compose.images.yml.4.4.1`. `git fetch --tags && git checkout v4.5.0`. 3. `make migrate-safe` from the 4.5.0 images: preflight passed (Alembic `a0f1a2b3c4d5`, 7 users, 2 campaigns); a second backup `questboard_manual_20261012T234701Z.dump`; 58 revisions to `d1054a2b3c4e`, the same backfill numbers as the rehearsal (parties 2, members 5, loot 4, custody 5; quests as pages 2). 133 seconds including the image pull. The 4.4.1 services kept running throughout. 4. `docker compose pull && docker compose up -d`: all seven services healthy on the 4.5.0 images within about a minute. **Verified** - `/ready` → `{"status":"ok"}`; `/api/meta` → `{"version":"4.5.0"}`, from inside the stack and at https://questboard.rhoving.com/api/meta . - The bot: "Version handshake OK: backend 4.5.0, bot API contract v2", connected to Discord and logged in. - No error, traceback, refusal or 5xx in the backend, worker, beat, bot or frontend logs in the first minutes; no "cannot be decrypted" line (prod's `SECRET_KEY` is unchanged). - Counts: 7 users, 2 campaigns, 22 sessions (none without a number), 58 wiki pages (56 + the 2 quests), 2 parties, 6 sheet definitions, both campaigns have an owner. **Not verified by me** - Nothing was looked at in a browser on prod, and nobody has signed in since the upgrade. - No recording, transcription, summary, reminder, email or Discord post has run on 4.5.0 yet. - Settings named in 4.5.0's `.env.example` that prod's `.env` does not set, so their defaults apply: `BOT_EVENTS_MAXLEN`, `ENVIRONMENT`, `MEDIA_ROOT`, `MEDIA_URL`, `POSTGRES_APP_USER`, `RCLONE_CONFIG_PATH`, `VITE_ALLOWED_HOSTS`. The app started and passed its own configuration checks; worth a read by the owner, `ENVIRONMENT` especially. **The owner's, on prod:** lane 10's Discord and email pieces (posts and reactions, DMs and slash commands, the emails, stop-these-emails, answering from an email, the calendar invite), a sign-in, and the starter content packs for character sheets. **If it had to be undone:** the two dumps above are from before the migrations; going back means restoring one, checking out `v4.4.1`, putting `docker-compose.images.yml.4.4.1` back as `docker-compose.images.yml`, and `docker compose up -d`. Anything written after the upgrade would be lost.

claude-bot closed this issue

2026-10-12 23:51:33 +00:00