rbrooks/Quest-Board

Watch

Fork 0

deploy: make the container restart policy per-host via RESTART_POLICY (#306) #481

Merged

claude-bot merged 1 commit from fix/306-restart-on-failure into main

2026-09-01 17:07:46 +00:00

claude-bot commented

2026-09-01 16:58:02 +00:00

Contributor

Closes #306.

Seven long-running services were hardcoded restart: unless-stopped. This closes the drift that let that keep coming back.

One deviation from what the issue asked for

#306 says to change the seven services to restart: on-failure. I've made it restart: ${RESTART_POLICY:-unless-stopped} instead, and here is why.

The issue's own "does this lose anything?" answer is "Not on this host" — and that is right, but this file ships to every install, not just docker-host. on-failure means the stack does not come back after a host reboot. That is correct where a boot sequencer owns startup ordering, and wrong for a plain single-stack install, where it would silently stop the stack returning after a power cut. "Safe for other people to run" is an explicit v4.x goal, so baking a host-specific policy into the shipped default seemed like the wrong trade.

Parameterising gets both:

Unset → unless-stopped. No existing install changes behaviour.
RESTART_POLICY=on-failure in .env → the seven services take it, which is what docker-host needs.
migrate stays "no" either way — one-shot, should not restart.

Host-specific config lives in that host's .env, which is where it belongs. Rhoving/iac-repo#296's playbook check is unaffected: it reads the running containers, so it catches drift whichever way the value arrived.

The drift had already happened

#306 states the containers "have been flipped live with docker update --restart on-failure, so the current boot is protected". That was no longer true when I checked. All seven were back on unless-stopped on docker-host — undone by a compose up -d at some point since 2026-08-12, which is precisely the failure mode the issue predicted.

So prod was not protected against the boot race at the time of writing, not merely at some future deploy.

Why the reasoning is in the file

#306 notes the rationale "is not obvious from this repo", and it isn't — it comes from iac-repo and a specific host's disk topology. The compose header now carries it, as does .env.example, so the next person to see on-failure and think "that looks wrong" has the answer in front of them rather than having to find iac-repo#279.

Verification

docker compose config parses and resolves correctly both ways:

	result
`RESTART_POLICY` unset	7 × `unless-stopped`, `migrate` `"no"`
`RESTART_POLICY=on-failure`	7 × `on-failure`, `migrate` `"no"`

Both exit 0. No code changes, so no test or lint impact.

Host state — already applied

Both done on docker-host ahead of this merge, so the deploy of this change is a no-op rather than a fix:

RESTART_POLICY=on-failure appended to /home/ryan/Quest-Board/.env (file kept at mode 600).
The seven running containers re-flipped with docker update --restart on-failure; migrate left on "no". Non-disruptive — docker compose ps still reported "Up 3 days (healthy)" for all seven afterwards, so nothing cycled.

dev (docker-test) deliberately gets nothing. It has no staggered-stacks unit — verified — so the whole rationale is absent there and it should keep unless-stopped so it returns after a reboot. Its containers are correctly on the default.

Once this merges, compose up -d on docker-host reads that .env and keeps the policy instead of reverting it, which is the loop #306 exists to close.

🤖 Generated with Claude Code

Closes #306. Seven long-running services were hardcoded `restart: unless-stopped`. This closes the drift that let that keep coming back. ## One deviation from what the issue asked for #306 says to change the seven services to `restart: on-failure`. I've made it `restart: ${RESTART_POLICY:-unless-stopped}` instead, and here is why. The issue's own "does this lose anything?" answer is *"Not on this host"* — and that is right, but this file ships to **every** install, not just docker-host. `on-failure` means the stack does not come back after a host reboot. That is correct where a boot sequencer owns startup ordering, and wrong for a plain single-stack install, where it would silently stop the stack returning after a power cut. "Safe for other people to run" is an explicit v4.x goal, so baking a host-specific policy into the shipped default seemed like the wrong trade. Parameterising gets both: - Unset → `unless-stopped`. No existing install changes behaviour. - `RESTART_POLICY=on-failure` in `.env` → the seven services take it, which is what docker-host needs. - `migrate` stays `"no"` either way — one-shot, should not restart. Host-specific config lives in that host's `.env`, which is where it belongs. Rhoving/iac-repo#296's playbook check is unaffected: it reads the *running containers*, so it catches drift whichever way the value arrived. ## The drift had already happened #306 states the containers "have been flipped **live** with `docker update --restart on-failure`, so the current boot is protected". **That was no longer true when I checked.** All seven were back on `unless-stopped` on docker-host — undone by a `compose up -d` at some point since 2026-08-12, which is precisely the failure mode the issue predicted. So prod was not protected against the boot race at the time of writing, not merely at some future deploy. ## Why the reasoning is in the file #306 notes the rationale "is not obvious from this repo", and it isn't — it comes from `iac-repo` and a specific host's disk topology. The compose header now carries it, as does `.env.example`, so the next person to see `on-failure` and think "that looks wrong" has the answer in front of them rather than having to find iac-repo#279. ## Verification `docker compose config` parses and resolves correctly both ways: | | result | |---|---| | `RESTART_POLICY` unset | 7 × `unless-stopped`, `migrate` `"no"` | | `RESTART_POLICY=on-failure` | 7 × `on-failure`, `migrate` `"no"` | Both exit 0. No code changes, so no test or lint impact. ## Host state — already applied Both done on docker-host ahead of this merge, so the deploy of this change is a no-op rather than a fix: - `RESTART_POLICY=on-failure` appended to `/home/ryan/Quest-Board/.env` (file kept at mode 600). - The seven running containers re-flipped with `docker update --restart on-failure`; `migrate` left on `"no"`. Non-disruptive — `docker compose ps` still reported "Up 3 days (healthy)" for all seven afterwards, so nothing cycled. **dev (docker-test) deliberately gets nothing.** It has no `staggered-stacks` unit — verified — so the whole rationale is absent there and it should keep `unless-stopped` so it returns after a reboot. Its containers are correctly on the default. Once this merges, `compose up -d` on docker-host reads that `.env` and keeps the policy instead of reverting it, which is the loop #306 exists to close. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit

2026-09-01 16:58:03 +00:00

deploy: make the container restart policy per-host via RESTART_POLICY (#306 )

CI / Bot/backend version sync (pull_request) Successful in 46s

Details

CI / Backend lint (ruff) (pull_request) Successful in 52s

Details

CI / Docker image build (pull_request) Successful in 17s

Details

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 1m25s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 2m3s

Details

CI / Bot tests and audit (pull_request) Successful in 2m15s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 8m37s

Details

b742b65b67

Seven long-running services were hardcoded `restart: unless-stopped`. On
docker-host that is actively harmful: the host runs ~16 Compose stacks on an
8-wide raidz1 of 7200 rpm HDDs, and starting them all at once at boot saturates
the pool and stalls the ext4 journal — which is what took docker-host and
Forgejo down on 2026-07-06. The fix there is `staggered-stacks`, a systemd unit
pacing stack startup, and it only works if dockerd starts nothing itself. A
container on `unless-stopped` is restored by dockerd before the paced sequence
reaches it, so the stack opts itself out of the mechanism protecting it.

The containers were already flipped live with `docker update`, which does not
touch the compose file — so the next `compose up` would have silently restored
`unless-stopped`. This closes that drift.

Deliberately a variable rather than the hardcoded `on-failure` the issue asked
for. This file ships to every install, and `on-failure` means the stack does
NOT come back after a host reboot — correct for a host with a boot sequencer,
wrong for a plain single-stack install, and "safe for other people to run" is
an explicit v4.x goal. Unset, the default is `unless-stopped`, so no existing
install changes behaviour; the host that needs `on-failure` opts in through its
own `.env`, which is where host-specific config belongs.

`migrate` stays on `"no"` in both cases — one-shot, should not restart.

The reasoning is written into the compose header and `.env.example` because it
originates outside this repo and is not inferable from here. Rhoving/iac-repo
 #296's playbook check still catches drift, since it reads the running
containers rather than this file.

Verified `docker compose config` parses and resolves correctly both ways:
unset → 7x unless-stopped, RESTART_POLICY=on-failure → 7x on-failure, migrate
"no" in both.

NOTE: docker-host's .env needs `RESTART_POLICY=on-failure` added, or the next
deploy reverts it to unless-stopped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

claude-bot merged commit 2c1885a166 into main

2026-09-01 17:07:46 +00:00

claude-bot referenced this pull request from a commit

2026-09-01 17:07:48 +00:00

Merge PR #481: make the container restart policy per-host via RESTART_POLICY (#306)

claude-bot referenced this pull request

2026-09-01 17:08:00 +00:00

deploy: set restart: on-failure in docker-compose.yml — unless-stopped bypasses docker-host's boot pacing #306