rbrooks/Quest-Board

Watch

Fork 0

deploy: set `restart: on-failure` in docker-compose.yml — `unless-stopped` bypasses docker-host's boot pacing #306

New issue

Closed

opened 2026-08-12 18:02:27 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-08-12 18:02:27 +00:00

Collaborator

Raised from Rhoving/iac-repo #279. One-line-per-service change; the reason is not obvious from this repo.

Context

Quest-Board's stack runs on docker-host (VM 110), which sits on an 8-wide raidz1 of 7200 rpm HDDs — roughly single-disk random IOPS shared by ~16 Compose stacks. Starting them all at once at boot saturates the pool and stalls the ext4 journal; that is what took docker-host and Forgejo down on 2026-07-06.

The fix there is staggered-stacks, a systemd unit that starts each stack one at a time, paced to disk I/O. It only works if dockerd starts nothing by itself — the unit must be the sole boot authority.

A container with restart: unless-stopped or always is restored by dockerd as soon as the daemon starts, before the paced sequence reaches it. The stack opts itself out of the pacing and races the mechanism meant to protect it.

What's needed

Seven Quest-Board containers were on unless-stopped:

quest-board-db-1       quest-board-redis-1    quest-board-backend-1
quest-board-frontend-1 quest-board-worker-1   quest-board-beat-1
quest-board-bot-1

They have been flipped live with docker update --restart on-failure, so the current boot is protected. docker update does not touch the compose file, so the next docker compose up -d from this repo silently restores unless-stopped.

Change restart: unless-stopped to restart: on-failure for those services in docker-compose.yml / docker-compose.override.yml.

Leave quest-board-migrate-1 alone — it is correctly on no, being a one-shot migration container. It should not restart at all.

Does this lose anything?

Not on this host. on-failure still restarts a container that crashes. The only behaviour given up is "come back automatically when the Docker daemon starts", which is precisely what is being removed on purpose — staggered-stacks provides it in a paced form instead.

Verification

Rhoving/iac-repo PR #296 adds a check to docker-host-ansible that reads the staggered-stacks manifest and fails if any container in it is on always/unless-stopped. Once merged, drift here surfaces as a red playbook run rather than a slow boot nobody attributes correctly.

Refs Rhoving/iac-repo#279, Rhoving/iac-repo#84, Rhoving/iac-repo#296

Raised from `Rhoving/iac-repo` #279. One-line-per-service change; the reason is not obvious from this repo. ## Context Quest-Board's stack runs on **docker-host** (VM 110), which sits on an 8-wide raidz1 of 7200 rpm HDDs — roughly single-disk random IOPS shared by ~16 Compose stacks. Starting them all at once at boot saturates the pool and stalls the ext4 journal; that is what took docker-host and Forgejo down on 2026-07-06. The fix there is `staggered-stacks`, a systemd unit that starts each stack one at a time, paced to disk I/O. It only works if **dockerd starts nothing by itself** — the unit must be the sole boot authority. A container with `restart: unless-stopped` or `always` is restored by dockerd as soon as the daemon starts, *before* the paced sequence reaches it. The stack opts itself out of the pacing and races the mechanism meant to protect it. ## What's needed Seven Quest-Board containers were on `unless-stopped`: ``` quest-board-db-1 quest-board-redis-1 quest-board-backend-1 quest-board-frontend-1 quest-board-worker-1 quest-board-beat-1 quest-board-bot-1 ``` They have been flipped **live** with `docker update --restart on-failure`, so the current boot is protected. `docker update` does not touch the compose file, so **the next `docker compose up -d` from this repo silently restores `unless-stopped`**. Change `restart: unless-stopped` to `restart: on-failure` for those services in `docker-compose.yml` / `docker-compose.override.yml`. **Leave `quest-board-migrate-1` alone** — it is correctly on `no`, being a one-shot migration container. It should not restart at all. ## Does this lose anything? Not on this host. `on-failure` still restarts a container that crashes. The only behaviour given up is "come back automatically when the Docker daemon starts", which is precisely what is being removed on purpose — `staggered-stacks` provides it in a paced form instead. ## Verification `Rhoving/iac-repo` PR #296 adds a check to `docker-host-ansible` that reads the staggered-stacks manifest and fails if any container in it is on `always`/`unless-stopped`. Once merged, drift here surfaces as a red playbook run rather than a slow boot nobody attributes correctly. Refs Rhoving/iac-repo#279, Rhoving/iac-repo#84, Rhoving/iac-repo#296

claude-bot added this to the v4.1.0 - Data Durability & Recovery milestone

2026-08-25 20:39:56 +00:00

claude-bot referenced this issue

2026-08-25 20:45:22 +00:00

[Review] Full audit of the session lifecycle: prep, recording, transcription, summarization, lore — scoping v4.0.0 #319

rbrooks referenced this issue from a commit

2026-09-01 16:57:49 +00:00

deploy: make the container restart policy per-host via RESTART_POLICY (#306)

claude-bot referenced this issue from a pull request that will close it,

2026-09-01 16:58:02 +00:00

deploy: make the container restart policy per-host via RESTART_POLICY (#306) #481

claude-bot closed this issue

2026-09-01 17:07:46 +00:00

claude-bot referenced this issue from a commit

2026-09-01 17:07:48 +00:00

Merge PR #481: make the container restart policy per-host via RESTART_POLICY (#306)

claude-bot commented

2026-09-01 17:08:00 +00:00

Author

Collaborator

Done in PR #481, with one deviation and one correction to this issue.

The drift had already happened

This issue says the containers "have been flipped live with docker update --restart on-failure, so the current boot is protected". That was no longer true. All seven were back on unless-stopped on docker-host when I checked — undone by a compose up -d at some point since 2026-08-12, exactly as this issue predicted would happen.

So docker-host was not protected against the boot race, and had not been for some time. Worth knowing, because the issue reads as though only a future deploy was at risk.

Deviation: a variable, not a hardcoded `on-failure`

The seven services are now restart: ${RESTART_POLICY:-unless-stopped}.

The reasoning here — shared spindles, staggered-stacks owning boot order — is specific to docker-host, but docker-compose.yml ships to every install. Hardcoding on-failure would mean any other deployment silently stops coming back after a reboot, and "safe for other people to run" is an explicit v4.x goal. Unset, the default is unless-stopped and nothing changes for anyone; the host that needs on-failure opts in through its own .env, which is where host-specific config belongs.

iac-repo#296's check is unaffected either way — it reads the running containers, not this file.

Applied on the hosts

docker-host: RESTART_POLICY=on-failure appended to .env (kept at mode 600), and the seven running containers re-flipped with docker update. migrate left on "no". Non-disruptive — docker compose ps still showed "Up 3 days (healthy)" across all seven afterwards, so nothing cycled. The next compose up -d now preserves the policy rather than reverting it, which closes the loop.
docker-test (dev): deliberately unchanged. It has no staggered-stacks unit — verified — so the rationale does not apply and it should keep unless-stopped so it returns after a reboot.

Also recorded in the file

This issue notes the reason "is not obvious from this repo", so the compose header and .env.example now carry it: the disk topology, the 2026-07-06 outage, why on-failure rather than always, and why migrate stays "no". Someone reading on-failure and thinking it looks wrong now has the answer in front of them instead of needing to find iac-repo#279.

docker compose config verified to resolve correctly both ways: unset → 7 × unless-stopped, RESTART_POLICY=on-failure → 7 × on-failure, migrate "no" in both.

Done in PR #481, with one deviation and one correction to this issue. ## The drift had already happened This issue says the containers "have been flipped **live** with `docker update --restart on-failure`, so the current boot is protected". **That was no longer true.** All seven were back on `unless-stopped` on docker-host when I checked — undone by a `compose up -d` at some point since 2026-08-12, exactly as this issue predicted would happen. So docker-host was not protected against the boot race, and had not been for some time. Worth knowing, because the issue reads as though only a future deploy was at risk. ## Deviation: a variable, not a hardcoded `on-failure` The seven services are now `restart: ${RESTART_POLICY:-unless-stopped}`. The reasoning here — shared spindles, staggered-stacks owning boot order — is specific to docker-host, but `docker-compose.yml` ships to every install. Hardcoding `on-failure` would mean any other deployment silently stops coming back after a reboot, and "safe for other people to run" is an explicit v4.x goal. Unset, the default is `unless-stopped` and nothing changes for anyone; the host that needs `on-failure` opts in through its own `.env`, which is where host-specific config belongs. iac-repo#296's check is unaffected either way — it reads the running containers, not this file. ## Applied on the hosts - **docker-host**: `RESTART_POLICY=on-failure` appended to `.env` (kept at mode 600), and the seven running containers re-flipped with `docker update`. `migrate` left on `"no"`. Non-disruptive — `docker compose ps` still showed "Up 3 days (healthy)" across all seven afterwards, so nothing cycled. The next `compose up -d` now *preserves* the policy rather than reverting it, which closes the loop. - **docker-test (dev)**: deliberately unchanged. It has no `staggered-stacks` unit — verified — so the rationale does not apply and it should keep `unless-stopped` so it returns after a reboot. ## Also recorded in the file This issue notes the reason "is not obvious from this repo", so the compose header and `.env.example` now carry it: the disk topology, the 2026-07-06 outage, why `on-failure` rather than `always`, and why `migrate` stays `"no"`. Someone reading `on-failure` and thinking it looks wrong now has the answer in front of them instead of needing to find iac-repo#279. `docker compose config` verified to resolve correctly both ways: unset → 7 × `unless-stopped`, `RESTART_POLICY=on-failure` → 7 × `on-failure`, `migrate` `"no"` in both.