rbrooks/TeaLeaves

Watch

Fork 0

deploy: set `restart: on-failure` in the compose file — `unless-stopped` bypasses docker-host's boot pacing #152

New issue

Closed

opened 2026-08-12 18:02:39 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-08-12 18:02:39 +00:00

Contributor

Raised from Rhoving/iac-repo #279. One-line-per-service change; the reason is not obvious from this repo.

Context

TeaLeaves' stack runs on docker-host (VM 110), which sits on an 8-wide raidz1 of 7200 rpm HDDs — roughly single-disk random IOPS shared by ~16 Compose stacks. Starting them all at once at boot saturates the pool and stalls the ext4 journal; that is what took docker-host and Forgejo down on 2026-07-06.

The fix there is staggered-stacks, a systemd unit that starts each stack one at a time, paced to disk I/O. It only works if dockerd starts nothing by itself — the unit must be the sole boot authority.

A container with restart: unless-stopped or always is restored by dockerd as soon as the daemon starts, before the paced sequence reaches it. The stack opts itself out of the pacing and races the mechanism meant to protect it.

What's needed

Five TeaLeaves containers were on unless-stopped:

tealeaves-db-1    tealeaves-redis-1    tealeaves-api-1
tealeaves-app-1   tealeaves-watchtower-1

They have been flipped live with docker update --restart on-failure, so the current boot is protected. docker update does not touch the compose file, so the next docker compose up -d from this repo silently restores unless-stopped.

Change restart: unless-stopped to restart: on-failure for those services in the deployed compose file.

Does this lose anything?

Not on this host. on-failure still restarts a container that crashes. The only behaviour given up is "come back automatically when the Docker daemon starts", which is precisely what is being removed on purpose — staggered-stacks provides it in a paced form instead.

Verification

Rhoving/iac-repo PR #296 adds a check to docker-host-ansible that reads the staggered-stacks manifest and fails if any container in it is on always/unless-stopped. Once merged, drift here surfaces as a red playbook run rather than a slow boot nobody attributes correctly.

Refs Rhoving/iac-repo#279, Rhoving/iac-repo#84, Rhoving/iac-repo#296

Raised from `Rhoving/iac-repo` #279. One-line-per-service change; the reason is not obvious from this repo. ## Context TeaLeaves' stack runs on **docker-host** (VM 110), which sits on an 8-wide raidz1 of 7200 rpm HDDs — roughly single-disk random IOPS shared by ~16 Compose stacks. Starting them all at once at boot saturates the pool and stalls the ext4 journal; that is what took docker-host and Forgejo down on 2026-07-06. The fix there is `staggered-stacks`, a systemd unit that starts each stack one at a time, paced to disk I/O. It only works if **dockerd starts nothing by itself** — the unit must be the sole boot authority. A container with `restart: unless-stopped` or `always` is restored by dockerd as soon as the daemon starts, *before* the paced sequence reaches it. The stack opts itself out of the pacing and races the mechanism meant to protect it. ## What's needed Five TeaLeaves containers were on `unless-stopped`: ``` tealeaves-db-1 tealeaves-redis-1 tealeaves-api-1 tealeaves-app-1 tealeaves-watchtower-1 ``` They have been flipped **live** with `docker update --restart on-failure`, so the current boot is protected. `docker update` does not touch the compose file, so **the next `docker compose up -d` from this repo silently restores `unless-stopped`**. Change `restart: unless-stopped` to `restart: on-failure` for those services in the deployed compose file. ## Does this lose anything? Not on this host. `on-failure` still restarts a container that crashes. The only behaviour given up is "come back automatically when the Docker daemon starts", which is precisely what is being removed on purpose — `staggered-stacks` provides it in a paced form instead. ## Verification `Rhoving/iac-repo` PR #296 adds a check to `docker-host-ansible` that reads the staggered-stacks manifest and fails if any container in it is on `always`/`unless-stopped`. Once merged, drift here surfaces as a red playbook run rather than a slow boot nobody attributes correctly. Refs Rhoving/iac-repo#279, Rhoving/iac-repo#84, Rhoving/iac-repo#296

rbrooks referenced this issue from a commit

2026-08-17 17:32:23 +00:00

fix(deploy): use restart: on-failure so staggered-stacks stays the boot authority

rbrooks closed this issue

2026-08-17 17:32:23 +00:00

claude-bot referenced this issue

2026-08-17 17:34:18 +00:00

deploy: nothing syncs deploy/compose.yaml to the hosts — both are a month stale, and #117's worker split isn't deployed anywhere #155

claude-bot commented

2026-08-17 17:34:32 +00:00

Author

Contributor

Done in fc8fb81 — but with two corrections to the issue as filed, and one caveat that means this isn't yet effective on prod.

Six services, not five

The issue lists five containers. deploy/compose.yaml has had a sixth since #117 split the BullMQ workers out of the API into a worker service, and it was on unless-stopped too. All six are now on-failure:

db  redis  api  worker  app  watchtower

The worker container doesn't exist on either host yet (see the caveat), but it will the moment the deployed compose file catches up, and it would have arrived on unless-stopped.

Rationale recorded in-repo

The issue notes the reason "is not obvious from this repo" — so it's now a comment block at the top of deploy/compose.yaml and a bullet in deploy/README.md, both pointing back at iac-repo#84/#279/#296. Anyone who reaches for unless-stopped here should hit the explanation before they change it.

Caveat: this does not protect prod yet

Two reasons, neither fixable in this repo:

1. staggered-stacks boots the wrong stack. /etc/staggered-stacks/order.conf has tealeaves|/home/ryan/TeaLeaves — the retired source dir, not /home/ryan/tealeaves-deploy. The file I just fixed is not the one the boot path runs. /home/ryan/TeaLeaves/docker-compose.yml is still on unless-stopped, so a successful boot from there would undo the live docker update flip regardless of this commit. Filed as Rhoving/iac-repo#318 — that also explains why prod has been down since 2026-08-11 (ERROR: tealeaves compose up failed, nothing on :8055 for six days).

2. Nothing syncs deploy/compose.yaml to the hosts. Both hosts run a Jul 17 copy with five services; the repo's has six. Committing here doesn't move it. Filed as #155.

I'm closing this because the change it asks for is committed and pushed. The two follow-ups are tracked above, and iac-repo#279's "commit restart: on-failure upstream" checkbox can be ticked for TeaLeaves.

Refs Rhoving/iac-repo#279, Rhoving/iac-repo#318, #155, #117

Done in `fc8fb81` — but with two corrections to the issue as filed, and one caveat that means this isn't yet effective on prod. ## Six services, not five The issue lists five containers. `deploy/compose.yaml` has had a sixth since #117 split the BullMQ workers out of the API into a `worker` service, and it was on `unless-stopped` too. All six are now `on-failure`: ``` db redis api worker app watchtower ``` The `worker` container doesn't exist on either host yet (see the caveat), but it will the moment the deployed compose file catches up, and it would have arrived on `unless-stopped`. ## Rationale recorded in-repo The issue notes the reason "is not obvious from this repo" — so it's now a comment block at the top of `deploy/compose.yaml` and a bullet in `deploy/README.md`, both pointing back at iac-repo#84/#279/#296. Anyone who reaches for `unless-stopped` here should hit the explanation before they change it. ## Caveat: this does not protect prod yet Two reasons, neither fixable in this repo: **1. `staggered-stacks` boots the wrong stack.** `/etc/staggered-stacks/order.conf` has `tealeaves|/home/ryan/TeaLeaves` — the retired source dir, not `/home/ryan/tealeaves-deploy`. The file I just fixed is not the one the boot path runs. `/home/ryan/TeaLeaves/docker-compose.yml` is still on `unless-stopped`, so a successful boot from there would undo the live `docker update` flip regardless of this commit. Filed as Rhoving/iac-repo#318 — that also explains why **prod has been down since 2026-08-11** (`ERROR: tealeaves compose up failed`, nothing on `:8055` for six days). **2. Nothing syncs `deploy/compose.yaml` to the hosts.** Both hosts run a Jul 17 copy with five services; the repo's has six. Committing here doesn't move it. Filed as #155. I'm closing this because the change it asks for is committed and pushed. The two follow-ups are tracked above, and iac-repo#279's "commit `restart: on-failure` upstream" checkbox can be ticked for TeaLeaves. Refs Rhoving/iac-repo#279, Rhoving/iac-repo#318, #155, #117

rbrooks referenced this issue from a commit

2026-08-17 18:00:09 +00:00

refactor(deploy): hand the compose file to Ansible; prod stops auto-deploying

rbrooks referenced this issue from a commit

2026-08-17 18:00:46 +00:00

feat(tealeaves): Ansible deployment coverage; pin prod and fix the boot manifest

claude-bot referenced this issue from a commit

2026-08-17 18:27:15 +00:00

Merge pull request 'feat(tealeaves): Ansible deployment coverage; pin prod and fix the boot manifest (#319)' (#319) from feat/tealeaves-ansible into main

claude-bot commented

2026-08-17 18:44:23 +00:00

Author

Contributor

Resolved — but in iac-repo, not here, and worth recording why.

This issue asked for restart: on-failure in the deployed compose file. The premise was that the file lived in this repo. It effectively no longer does: Rhoving/iac-repo PR #319 moved the deployed topology into tealeaves-ansible's compose template, and upstream deleted its own copy in the same change (8cd5ad5). So there is now exactly one place the restart policy exists, and it renders on-failure for every service.

Deployed to both hosts today. On docker-host:

/tealeaves-worker-1 on-failure
/tealeaves-app-1    on-failure
/tealeaves-api-1    on-failure
/tealeaves-db-1     on-failure
/tealeaves-redis-1  on-failure

The staggered-stacks verify task agrees: "Every container in /etc/staggered-stacks/order.conf is on on-failure — dockerd will start nothing at boot and staggered-stacks remains the sole authority."

The drift this issue warned about — that docker update --restart doesn't touch the compose file, so the next compose up -d silently restores unless-stopped — is now closed off structurally: the file is generated from the role, so a run reasserts on-failure rather than undoing it.

Worth noting the wider context, since it is the reason this sat open: the compose file on both hosts had been hand-copied once in July and never updated, so this fix had nothing to reach even after it was written. The same staleness meant the worker service from #117 was never actually running on either host until today. See Rhoving/iac-repo#318 for the six-day outage that surfaced all of it.

Closing.

Resolved — but in `iac-repo`, not here, and worth recording why. This issue asked for `restart: on-failure` in the deployed compose file. The premise was that the file lived in this repo. It effectively no longer does: `Rhoving/iac-repo` PR #319 moved the deployed topology into `tealeaves-ansible`'s compose template, and upstream deleted its own copy in the same change (8cd5ad5). So there is now exactly one place the restart policy exists, and it renders `on-failure` for every service. Deployed to both hosts today. On docker-host: ``` /tealeaves-worker-1 on-failure /tealeaves-app-1 on-failure /tealeaves-api-1 on-failure /tealeaves-db-1 on-failure /tealeaves-redis-1 on-failure ``` The staggered-stacks verify task agrees: *"Every container in /etc/staggered-stacks/order.conf is on on-failure — dockerd will start nothing at boot and staggered-stacks remains the sole authority."* The drift this issue warned about — that `docker update --restart` doesn't touch the compose file, so the next `compose up -d` silently restores `unless-stopped` — is now closed off structurally: the file is generated from the role, so a run reasserts `on-failure` rather than undoing it. Worth noting the wider context, since it is the reason this sat open: the compose file on both hosts had been hand-copied once in July and never updated, so this fix had nothing to reach even after it was written. The same staleness meant the `worker` service from #117 was **never actually running on either host** until today. See `Rhoving/iac-repo`#318 for the six-day outage that surfaced all of it. Closing.

claude-bot referenced this issue

2026-08-17 19:35:13 +00:00

deploy: nothing syncs deploy/compose.yaml to the hosts — both are a month stale, and #117's worker split isn't deployed anywhere #155