make migrate-safe migrates with the running release's code, not the one being installed #1045

New issue

Closed

opened 2026-10-03 14:13:29 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-10-03 14:13:29 +00:00

Collaborator

Found while writing the image-host update procedure (#1043, PR #1044).

What happens

make migrate-safe is docker compose exec backend … three times: db_preflight, backup_now, then alembic upgrade head. exec runs in the currently running backend container. The backend-prod image bakes the code in, so after a git pull (source build) or a git checkout and pull (image host), that container is still the old release. Its alembic upgrade head therefore knows only the old release's revisions and applies nothing new. The new migrations then run unguarded in the migrate service on up, which is exactly what migrate-safe exists to avoid.

docs/OPERATIONS.md still tells a source-build self-hoster to run it "before bringing up the new stack" for a major upgrade, and lists "use make migrate-safe before production migrations" under the safety rules.

Already done

PR #1044 stops recommending it for image hosts. They run the read-only db_preflight from the new image instead (docker compose run --rm migrate python -m app.cli db_preflight), after make backup-now.

To decide

Either make the target build or pull first and run its steps from the new image (docker compose run --rm migrate …), or retire it in favour of make backup-now plus that preflight, and correct the install section and the safety rules either way.

Found while writing the image-host update procedure (#1043, PR #1044). ## What happens `make migrate-safe` is `docker compose exec backend …` three times: `db_preflight`, `backup_now`, then `alembic upgrade head`. `exec` runs in the **currently running** `backend` container. The `backend-prod` image bakes the code in, so after a `git pull` (source build) or a `git checkout` and `pull` (image host), that container is still the old release. Its `alembic upgrade head` therefore knows only the old release's revisions and applies nothing new. The new migrations then run unguarded in the `migrate` service on `up`, which is exactly what `migrate-safe` exists to avoid. `docs/OPERATIONS.md` still tells a source-build self-hoster to run it "before bringing up the new stack" for a major upgrade, and lists "use `make migrate-safe` before production migrations" under the safety rules. ## Already done PR #1044 stops recommending it for image hosts. They run the read-only `db_preflight` from the new image instead (`docker compose run --rm migrate python -m app.cli db_preflight`), after `make backup-now`. ## To decide Either make the target build or pull first and run its steps from the new image (`docker compose run --rm migrate …`), or retire it in favour of `make backup-now` plus that preflight, and correct the install section and the safety rules either way.

claude-bot referenced this issue

2026-10-03 15:39:35 +00:00

Prereleases for testing on dev, and both hosts run the published images #1043

claude-bot commented

2026-10-04 00:03:32 +00:00

Author

Collaborator

Owner's answer to Q146 (O30), 2026-10-04: (a), as recommended. Fix the command so it runs from the new release: pull or build first, then run the preflight, the backup and the migration from the new image, not inside the old running container. The two places in the operations guide that still recommend the old behaviour get corrected with it. To be recorded as D274. Being built now.

**Owner's answer to Q146 (O30), 2026-10-04: (a), as recommended.** Fix the command so it runs from the new release: pull or build first, then run the preflight, the backup and the migration from the new image, not inside the old running container. The two places in the operations guide that still recommend the old behaviour get corrected with it. To be recorded as D274. Being built now.

claude-bot referenced this issue

2026-10-04 00:40:45 +00:00

Decisions D274 to D278: the owner's answers of 4 October (Q146 to Q150) #1060

rbrooks referenced this issue from a commit

2026-10-04 01:37:57 +00:00

make migrate-safe runs from the new release's image (#1045)

rbrooks referenced this issue from a commit

2026-10-04 01:59:46 +00:00

Format the migrate-safe detection test (#1045)

claude-bot referenced this issue from a pull request that will close it,

2026-10-04 02:00:50 +00:00

make migrate-safe runs from the new release's image (#1045) #1062

claude-bot closed this issue

2026-10-04 02:27:08 +00:00

claude-bot referenced this issue from a commit

2026-10-04 02:27:08 +00:00

Merge PR #1062: make migrate-safe runs from the new release's image (#1045)

claude-bot commented

2026-10-04 02:27:08 +00:00

Author

Collaborator

Done in PR #1062, merged to main on 2026-10-04 with CI green. D274 is recorded (PR #1060).

make migrate-safe now calls scripts/migrate-safe.sh. The script fetches the new release (pull on an image host, build on a source build, asked of compose itself), then runs the preflight, the backup and the migration from it in throwaway containers. No running service is touched, and it stops at the first failure and names the step. docs/OPERATIONS.md, the compose comment and the project profile recommend it again.

Proved on docker-test in a throwaway stack: both modes, and the failure path with the database stopped. The running backend's container id and start time were unchanged.

Not seen: a real network pull of a newer tag. The first production update to 4.5.0 will be that.

Done in PR #1062, merged to main on 2026-10-04 with CI green. D274 is recorded (PR #1060). `make migrate-safe` now calls `scripts/migrate-safe.sh`. The script fetches the new release (pull on an image host, build on a source build, asked of compose itself), then runs the preflight, the backup and the migration from it in throwaway containers. No running service is touched, and it stops at the first failure and names the step. `docs/OPERATIONS.md`, the compose comment and the project profile recommend it again. **Proved** on docker-test in a throwaway stack: both modes, and the failure path with the database stopped. The running backend's container id and start time were unchanged. **Not seen:** a real network pull of a newer tag. The first production update to 4.5.0 will be that.