rbrooks/TeaLeaves

Watch

Fork 0

deploy: nothing syncs deploy/compose.yaml to the hosts — both are a month stale, and #117's worker split isn't deployed anywhere #155

New issue

Closed

opened 2026-08-17 17:34:18 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-08-17 17:34:18 +00:00

Contributor

deploy/compose.yaml is described in CLAUDE.md as "the source of truth for that compose file", but there is no mechanism that gets it onto a host. It was hand-copied once at cutover and has not moved since.

Current state

	repo	ai-dev-server	docker-host
size	4579 B	3475 B	3475 B
mtime	—	Jul 17 21:25	Jul 17 21:39
services	6	5	5
`worker`	yes	no	no

Both hosts run the pre-#117 file. So the worker split — the whole point of #117 — is not in effect on either host: RUN_WORKERS_IN_API is unset, the image defaults it to true, and background jobs still run inside the API process. That is a working configuration, just not the one the repo and deploy/README.md describe, and nobody would notice from either side.

The same gap is what makes #152 a no-op in practice: the restart-policy flip is now correct in the repo, and neither host will ever see it.

Discovered while diagnosing the prod outage (Rhoving/iac-repo#318) — separate root cause, same underlying "the deployed artifact drifts from the repo silently" shape.

Why the image pipeline doesn't cover this

Watchtower swaps images, not topology. deploy/README.md is explicit about it: "an image auto-update alone never changes topology". Anything expressed in the compose file — a new service, an env var, a restart policy, a port, a volume — needs the file itself to move. Today that means someone remembering to scp it, on two hosts, with no signal when they forget.

Options

A. Ansible-managed, like the manifest (recommended)
Add a tealeaves role (or extend docker-host-ansible) that templates compose.yaml into ~/tealeaves-deploy/ on both hosts from the repo, with TEALEAVES_CHANNEL/APP_PORT as host vars, and docker compose up -d on change. Pros: same mechanism, same review path and drift-detection as staggered-stacks itself — and iac-repo#318 is about to touch that role anyway, so the two land together. .env stays hand-managed (secrets). Cons: a TeaLeaves change now needs an iac-repo run to reach the hosts; the two repos have to stay in step.

B. CI publishes the compose file as a release asset; host pulls it
release-images.yml attaches compose.yaml to each release; a small host-side unit fetches the one matching its channel before compose up. Pros: keeps the file's lifecycle attached to the release that needs it, so "this version needs a new service" travels with the version. Cons: new host-side machinery, and it races Watchtower — the image can land before the compose file does.

C. Make the hosts pull the repo
git clone the repo to each host, git pull in a timer, run compose from deploy/. Pros: trivially correct, no new tooling. Cons: puts a source checkout back on the hosts — which is precisely what caused iac-repo#318. Recommend against.

D. Do nothing; document the manual step
Add a "when you change deploy/compose.yaml, copy it to both hosts" line to the post-phase checklist. Honest and free, but it is the process we already have, and it has already failed once for a month.

Whichever way this goes, a drift check is worth having independently: compare the deployed file against the repo's and report. That is a smaller piece of work than any of the above and would have caught this in July.

Immediate, regardless of option

Copy the current deploy/compose.yaml to ~/tealeaves-deploy/ on both hosts and re-up, so #152 and #117 are actually deployed
Decide between A–D

Refs #152, #117, #134, Rhoving/iac-repo#318

`deploy/compose.yaml` is described in CLAUDE.md as "the source of truth for that compose file", but there is no mechanism that gets it onto a host. It was hand-copied once at cutover and has not moved since. ## Current state | | repo | ai-dev-server | docker-host | |---|---|---|---| | size | 4579 B | 3475 B | 3475 B | | mtime | — | Jul 17 21:25 | Jul 17 21:39 | | services | 6 | 5 | 5 | | `worker` | yes | **no** | **no** | Both hosts run the pre-#117 file. So the worker split — the whole point of #117 — is **not in effect on either host**: `RUN_WORKERS_IN_API` is unset, the image defaults it to `true`, and background jobs still run inside the API process. That is a working configuration, just not the one the repo and `deploy/README.md` describe, and nobody would notice from either side. The same gap is what makes #152 a no-op in practice: the restart-policy flip is now correct in the repo, and neither host will ever see it. Discovered while diagnosing the prod outage (Rhoving/iac-repo#318) — separate root cause, same underlying "the deployed artifact drifts from the repo silently" shape. ## Why the image pipeline doesn't cover this Watchtower swaps **images**, not topology. `deploy/README.md` is explicit about it: "an image auto-update alone never changes topology". Anything expressed in the compose file — a new service, an env var, a restart policy, a port, a volume — needs the file itself to move. Today that means someone remembering to `scp` it, on two hosts, with no signal when they forget. ## Options **A. Ansible-managed, like the manifest** *(recommended)* Add a `tealeaves` role (or extend `docker-host-ansible`) that templates `compose.yaml` into `~/tealeaves-deploy/` on both hosts from the repo, with `TEALEAVES_CHANNEL`/`APP_PORT` as host vars, and `docker compose up -d` on change. Pros: same mechanism, same review path and drift-detection as `staggered-stacks` itself — and iac-repo#318 is about to touch that role anyway, so the two land together. `.env` stays hand-managed (secrets). Cons: a TeaLeaves change now needs an iac-repo run to reach the hosts; the two repos have to stay in step. **B. CI publishes the compose file as a release asset; host pulls it** `release-images.yml` attaches `compose.yaml` to each release; a small host-side unit fetches the one matching its channel before `compose up`. Pros: keeps the file's lifecycle attached to the release that needs it, so "this version needs a new service" travels with the version. Cons: new host-side machinery, and it races Watchtower — the image can land before the compose file does. **C. Make the hosts pull the repo** `git clone` the repo to each host, `git pull` in a timer, run compose from `deploy/`. Pros: trivially correct, no new tooling. Cons: puts a source checkout back on the hosts — which is precisely what caused iac-repo#318. Recommend against. **D. Do nothing; document the manual step** Add a "when you change `deploy/compose.yaml`, copy it to both hosts" line to the post-phase checklist. Honest and free, but it is the process we already have, and it has already failed once for a month. Whichever way this goes, a **drift check** is worth having independently: compare the deployed file against the repo's and report. That is a smaller piece of work than any of the above and would have caught this in July. ## Immediate, regardless of option - [ ] Copy the current `deploy/compose.yaml` to `~/tealeaves-deploy/` on both hosts and re-up, so #152 and #117 are actually deployed - [ ] Decide between A–D Refs #152, #117, #134, Rhoving/iac-repo#318

claude-bot referenced this issue

2026-08-17 17:34:32 +00:00

deploy: set restart: on-failure in the compose file — unless-stopped bypasses docker-host's boot pacing #152

rbrooks referenced this issue from a commit

2026-08-17 18:00:09 +00:00

refactor(deploy): hand the compose file to Ansible; prod stops auto-deploying

rbrooks referenced this issue from a commit

2026-08-17 18:00:46 +00:00

feat(tealeaves): Ansible deployment coverage; pin prod and fix the boot manifest

claude-bot commented

2026-08-17 18:01:42 +00:00

Author

Contributor

Option A (Ansible-managed) chosen and implemented.

This repo: 8cd5ad5 — deploy/compose.yaml deleted, deploy/README.md rewritten to document the split, CLAUDE.md's post-phase checklist and release steps updated. Keeping a copy here would have relocated the drift rather than fixed it, so there is now exactly one place the deployed topology exists.
iac-repo: Rhoving/iac-repo#319 — new tealeaves-ansible/ project rendering compose.yaml + .env into /opt/tealeaves-svc on both hosts. Open for review; not merged, and no playbook has been run against either host.

One decision worth recording here because it changes this repo's release process: prod no longer auto-deploys. It runs an image tag pinned in iac-repo's group_vars and moves on an explicit playbook run that pulls, snapshots the database and gates on /health. Dev keeps the :beta Watchtower loop. Shipping a release is now tag here → bump the pin there → ansible-playbook site.yml -l docker-host.

That is a direct response to what this issue turned up: the auto-deploy prod was supposed to have never actually reached its api/app containers (Rhoving/iac-repo#318), so "prod is current" was an assumption nothing checked. A pinned tag plus a health-gated run makes it a question with an answer.

The drift check floated in the original write-up is now unnecessary for the compose file — there is only one copy — but the same reasoning still applies to the .env, which remains hand-managed for secrets.

Leaving this open until #319 merges and both hosts have actually been cut over.

Option **A (Ansible-managed)** chosen and implemented. - **This repo:** `8cd5ad5` — `deploy/compose.yaml` deleted, `deploy/README.md` rewritten to document the split, `CLAUDE.md`'s post-phase checklist and release steps updated. Keeping a copy here would have relocated the drift rather than fixed it, so there is now exactly one place the deployed topology exists. - **iac-repo:** [Rhoving/iac-repo#319](https://git.rhoving.com/Rhoving/iac-repo/pulls/319) — new `tealeaves-ansible/` project rendering `compose.yaml` + `.env` into `/opt/tealeaves-svc` on both hosts. Open for review; not merged, and no playbook has been run against either host. One decision worth recording here because it changes this repo's release process: **prod no longer auto-deploys.** It runs an image tag pinned in iac-repo's `group_vars` and moves on an explicit playbook run that pulls, snapshots the database and gates on `/health`. Dev keeps the `:beta` Watchtower loop. Shipping a release is now tag here → bump the pin there → `ansible-playbook site.yml -l docker-host`. That is a direct response to what this issue turned up: the auto-deploy prod was supposed to have never actually reached its api/app containers (Rhoving/iac-repo#318), so "prod is current" was an assumption nothing checked. A pinned tag plus a health-gated run makes it a question with an answer. The drift check floated in the original write-up is now unnecessary for the compose file — there is only one copy — but the same reasoning still applies to the `.env`, which remains hand-managed for secrets. Leaving this open until #319 merges and both hosts have actually been cut over.

claude-bot referenced this issue from a commit

2026-08-17 18:27:15 +00:00

Merge pull request 'feat(tealeaves): Ansible deployment coverage; pin prod and fix the boot manifest (#319)' (#319) from feat/tealeaves-ansible into main

claude-bot referenced this issue

2026-08-17 19:35:03 +00:00

worker container is permanently unhealthy — it inherits the API image's HTTP healthcheck but serves no HTTP #156

claude-bot commented

2026-08-17 19:35:13 +00:00

Author

Contributor

Deployed — both hosts cut over, verified

Rhoving/iac-repo#319 merged and the play has been run against both hosts.

docker-host   tealeaves  running(5)  /opt/tealeaves-svc/compose.yaml   {"status":"ok","db":"ok","redis":"ok"}
ai-dev-server tealeaves  running(6)  /opt/tealeaves-svc/compose.yaml   {"status":"ok","db":"ok","redis":"ok"}

Every item this issue raised is resolved:

One config file per host. docker compose ls shows a single compose.yaml per project — the split-stack state from Rhoving/iac-repo#318 is gone.
The worker service is deployed for the first time (5 containers on prod, 6 on dev — the difference is dev's Watchtower, by design).
Prod is on 9.4.0, up from the v7.2.0 it had been silently serving. The v7.2.0 → v9.4.0 migration applied cleanly.
#152's on-failure is live, on the file the boot path actually reads.

Closing.

One thing the cutover surfaced, filed as #156: the worker container is permanently unhealthy on both hosts. It runs the API image with a different command and so inherits that image's HTTP healthcheck, but serves no HTTP. The worker is functioning correctly — 11 workers up, nightly purge ran — so it is a false negative, not an outage. It was invisible until now precisely because of this issue: the worker container had never actually run anywhere.

## Deployed — both hosts cut over, verified `Rhoving/iac-repo#319` merged and the play has been run against both hosts. ``` docker-host tealeaves running(5) /opt/tealeaves-svc/compose.yaml {"status":"ok","db":"ok","redis":"ok"} ai-dev-server tealeaves running(6) /opt/tealeaves-svc/compose.yaml {"status":"ok","db":"ok","redis":"ok"} ``` Every item this issue raised is resolved: - **One config file per host.** `docker compose ls` shows a single `compose.yaml` per project — the split-stack state from Rhoving/iac-repo#318 is gone. - **The `worker` service is deployed** for the first time (5 containers on prod, 6 on dev — the difference is dev's Watchtower, by design). - **Prod is on 9.4.0**, up from the v7.2.0 it had been silently serving. The v7.2.0 → v9.4.0 migration applied cleanly. - **#152's `on-failure` is live**, on the file the boot path actually reads. Closing. One thing the cutover surfaced, filed as **#156**: the `worker` container is permanently `unhealthy` on both hosts. It runs the API image with a different command and so inherits that image's HTTP healthcheck, but serves no HTTP. The worker is functioning correctly — 11 workers up, nightly purge ran — so it is a false negative, not an outage. It was invisible until now precisely because of this issue: the worker container had never actually run anywhere.

claude-bot closed this issue

2026-08-17 19:35:26 +00:00

rbrooks referenced this issue from a commit

2026-08-17 19:42:18 +00:00

fix(worker): give the worker container a liveness probe it can actually pass