rbrooks/TeaLeaves

Watch

Fork 0

Auto-deploy: ship main to dev and tagged releases to prod (no more hand-deploys) #134

New issue

Closed

opened 2026-07-17 01:25:59 +00:00 by claude-bot · 3 comments

claude-bot commented

2026-07-17 01:25:59 +00:00

Contributor

Category: ops · Effort: M

Problem

Neither environment auto-deploys. Every deploy is the hand-run checklist in CLAUDE.md (steps 5–9): ssh in, git pull, docker compose up --build -d with the GIT_SHA/GIT_DESCRIBE stamp, then check logs/health by hand.

Consequences seen in practice:

Prod drifts silently. Before the v7.1.0 release (2026-07-16), prod (10.1.1.7) was still on v7.0.0 — a full release behind, for ~2 weeks. Nothing surfaced that.
Config drift goes unnoticed. Prod's docker-compose.yml had an uncommitted local edit (all four services restart: unless-stopped → on-failure, made 2026-07-09 with a .bak beside it) that no one was tracking. It survived because deploys are manual and nobody diffs the tree. It fought the v7.1.0 deploy (which edits that same file) and was only caught because the deploy collided with it. Reverted 2026-07-16 as stale.
The .forgejo/workflows/ci.yml header already anticipates this: "Image build/push is intentionally omitted: the Actions runner exposes no Docker daemon to jobs. Revisit alongside CD."

Proposal

Dev — deploy main on every green CI run. Low risk, gives the "does it boot" signal automatically instead of me driving it by hand each time.

Prod — deploy on a version tag (v*) only, after CI is green. Keeps prod deliberate (matching how releases are cut today) while removing the drift and the hand-typed GIT_SHA stamp.

Both must preserve what the manual checklist gets right:

stamp GIT_SHA / GIT_DESCRIBE so the admin version panel stays truthful;
verify /health after the deploy and fail loudly if it doesn't come back;
refuse to deploy over a dirty working tree rather than clobbering it — the prod drift above would have been silently destroyed by a naive git reset --hard. Surface it instead.

Open questions

Image build vs build-on-host. Cleanest CD builds and pushes an image, then the host just pulls a tag. That needs a registry and a runner that can build — the runner now has DinD (forgejo-runner VM 150), so this may already be viable; #103 (build the prod image in CI) is the natural prerequisite and should probably land first.
Push mechanism. Runner → host over SSH, or a pull-based agent on each host? The runner is on VLAN InternalProd (10.1.1.20); docker-host is 10.1.1.7, dev is 10.1.1.14 — needs a deploy key/service account with a tight blast radius, not a general-purpose root key.
Prod safety. Want a manual approval gate, or is "tag = intent to ship" enough?

⚠️ Rhoving/iac-repo#159 — CI fails intermittently with no code change. Auto-deploy must not be built on a flaky green signal; if CI is untrustworthy, gating deploys on it either blocks good releases or ships on a coin flip. Fix that first.
#103 — CI should build the production Docker image (prerequisite for a registry-based CD).
Runner capacity/config lives in Rhoving/iac-repo (forgejo-runner-ansible); the CD workflow itself lives here.

Acceptance criteria

A push to main with green CI deploys to dev automatically, with the version stamp intact, and fails the run if /health doesn't return {"status":"ok"}.
Pushing a v* tag with green CI deploys that exact tag to prod, stamped, with the same health gate.
A deploy aborts (and reports) instead of overwriting uncommitted changes on the target host.
CLAUDE.md's post-phase checklist is updated to describe the automated path and keep the manual steps only as the documented fallback.

**Category:** ops · **Effort:** M ## Problem Neither environment auto-deploys. Every deploy is the hand-run checklist in `CLAUDE.md` (steps 5–9): ssh in, `git pull`, `docker compose up --build -d` with the `GIT_SHA`/`GIT_DESCRIBE` stamp, then check logs/health by hand. Consequences seen in practice: - **Prod drifts silently.** Before the v7.1.0 release (2026-07-16), prod (10.1.1.7) was still on **v7.0.0** — a full release behind, for ~2 weeks. Nothing surfaced that. - **Config drift goes unnoticed.** Prod's `docker-compose.yml` had an **uncommitted local edit** (all four services `restart: unless-stopped` → `on-failure`, made 2026-07-09 with a `.bak` beside it) that no one was tracking. It survived because deploys are manual and nobody diffs the tree. It fought the v7.1.0 deploy (which edits that same file) and was only caught because the deploy collided with it. Reverted 2026-07-16 as stale. - The `.forgejo/workflows/ci.yml` header already anticipates this: *"Image build/push is intentionally omitted: the Actions runner exposes no Docker daemon to jobs. Revisit alongside CD."* ## Proposal **Dev — deploy `main` on every green CI run.** Low risk, gives the "does it boot" signal automatically instead of me driving it by hand each time. **Prod — deploy on a version tag (`v*`) only**, after CI is green. Keeps prod deliberate (matching how releases are cut today) while removing the drift and the hand-typed `GIT_SHA` stamp. Both must preserve what the manual checklist gets right: - stamp `GIT_SHA` / `GIT_DESCRIBE` so the admin version panel stays truthful; - verify `/health` after the deploy and fail loudly if it doesn't come back; - **refuse to deploy over a dirty working tree** rather than clobbering it — the prod drift above would have been silently destroyed by a naive `git reset --hard`. Surface it instead. ## Open questions - **Image build vs build-on-host.** Cleanest CD builds and pushes an image, then the host just pulls a tag. That needs a registry and a runner that can build — the runner now has DinD (`forgejo-runner` VM 150), so this may already be viable; #103 (build the prod image in CI) is the natural prerequisite and should probably land first. - **Push mechanism.** Runner → host over SSH, or a pull-based agent on each host? The runner is on VLAN InternalProd (10.1.1.20); docker-host is 10.1.1.7, dev is 10.1.1.14 — needs a deploy key/service account with a tight blast radius, not a general-purpose root key. - **Prod safety.** Want a manual approval gate, or is "tag = intent to ship" enough? ## Blocked on / related - ⚠️ **Rhoving/iac-repo#159** — CI fails intermittently with no code change. **Auto-deploy must not be built on a flaky green signal**; if CI is untrustworthy, gating deploys on it either blocks good releases or ships on a coin flip. Fix that first. - **#103** — CI should build the production Docker image (prerequisite for a registry-based CD). - Runner capacity/config lives in `Rhoving/iac-repo` (`forgejo-runner-ansible`); the CD workflow itself lives here. ## Acceptance criteria - [ ] A push to `main` with green CI deploys to dev automatically, with the version stamp intact, and fails the run if `/health` doesn't return `{"status":"ok"}`. - [ ] Pushing a `v*` tag with green CI deploys that exact tag to prod, stamped, with the same health gate. - [ ] A deploy aborts (and reports) instead of overwriting uncommitted changes on the target host. - [ ] `CLAUDE.md`'s post-phase checklist is updated to describe the automated path and keep the manual steps only as the documented fallback.

claude-bot added this to the v7.2.0 - Reliability, data integrity & docs milestone

2026-07-17 01:26:07 +00:00

claude-bot added the

enhancement

label

2026-07-17 01:26:23 +00:00

claude-bot referenced this issue

2026-07-17 03:19:15 +00:00

Nothing restarts the API when it goes unhealthy — Docker Compose does not act on healthcheck status #137

claude-bot commented

2026-07-17 03:20:58 +00:00

Author

Contributor

Status: still blocked, deliberately not started. Two of the three prerequisites are now done, and two open questions are resolved — recording that so whoever picks this up doesn't re-derive it.

Prerequisites

✅ #103 done (109c53a) — CI now builds both production images on every push, so a Dockerfile/peer-dep regression can't reach a deploy. The job also asserts the migrations copy landed in the image.
✅ #97 done (109c53a) — /health now actually probes DB + Redis and returns 503 when either is unreachable. This mattered more than it looked: the deploy gate in this issue's acceptance criteria was going to check /health, and until now /health returned {"status":"ok"} regardless of whether the database was up. An auto-deploy gated on the old endpoint would have cheerfully reported success over a broken instance.
❌ Rhoving/iac-repo#159 — CI flakes. Blocking. Second occurrence today (run 192, c1ed843): API job failed on a commit that passes tsc, passes 242/242 locally, passes against a from-scratch DB+Redis, and passed on re-run. Gating deploys on that signal means either good releases blocked at random, or — if the gate is made advisory to compensate — deploying on a coin flip. Both are worse than the current manual checklist.

Open questions — two now answered

Registry: solved, no external dependency. Forgejo's own container registry is live on this instance (https://git.rhoving.com/v2/ → 401, i.e. present and requiring auth; has_packages: true on the repo). So the image-based CD path is viable without standing anything up: CI builds → pushes to git.rhoving.com/rbrooks/tealeaves-api:<tag> → the host pulls that tag. Auth via a scoped Actions secret (see the forgejo-actions-secrets skill in iac-repo).
Can the runner build images: yes. The ci.yml comment claiming "the Actions runner exposes no Docker daemon to jobs" was stale — it described the old docker-host runner (removed 2026-07-01, iac-repo#32). The dedicated runner VM mounts the DinD socket into every job container on purpose (forgejo-runner-ansible: forgejo_runner_container_options: "-v /var/run/docker.sock:/var/run/docker.sock" — "so docker build works inside CI jobs"). Comment corrected in 109c53a; the new image job proves it in practice.
Still open: push mechanism (runner → host over SSH vs. a pull-based agent) and whether prod wants a manual approval gate or "tag = intent to ship".

Recommended sequence

Fix iac-repo#159 (or establish it's environmental and bound it).
Dev auto-deploy first — main on green CI. Low blast radius, and it exercises the whole path.
Prod on v* tags once dev has run clean for a while.

Not starting the implementation until (1) lands — building a deploy pipeline on a signal we currently can't trust would bake the flake into releases.

**Status: still blocked, deliberately not started.** Two of the three prerequisites are now done, and two open questions are resolved — recording that so whoever picks this up doesn't re-derive it. ### Prerequisites - ✅ **#103 done** (`109c53a`) — CI now builds both production images on every push, so a Dockerfile/peer-dep regression can't reach a deploy. The job also asserts the migrations copy landed in the image. - ✅ **#97 done** (`109c53a`) — `/health` now actually probes DB + Redis and returns 503 when either is unreachable. **This mattered more than it looked:** the deploy gate in this issue's acceptance criteria was going to check `/health`, and until now `/health` returned `{"status":"ok"}` regardless of whether the database was up. An auto-deploy gated on the old endpoint would have cheerfully reported success over a broken instance. - ❌ **Rhoving/iac-repo#159 — CI flakes.** Blocking. **Second occurrence today** (run 192, `c1ed843`): API job failed on a commit that passes `tsc`, passes 242/242 locally, passes against a from-scratch DB+Redis, and passed on re-run. Gating deploys on that signal means either good releases blocked at random, or — if the gate is made advisory to compensate — deploying on a coin flip. Both are worse than the current manual checklist. ### Open questions — two now answered - **Registry: solved, no external dependency.** Forgejo's own container registry is live on this instance (`https://git.rhoving.com/v2/` → `401`, i.e. present and requiring auth; `has_packages: true` on the repo). So the image-based CD path is viable without standing anything up: CI builds → pushes to `git.rhoving.com/rbrooks/tealeaves-api:<tag>` → the host pulls that tag. Auth via a scoped Actions secret (see the `forgejo-actions-secrets` skill in iac-repo). - **Can the runner build images: yes.** The `ci.yml` comment claiming "the Actions runner exposes no Docker daemon to jobs" was **stale** — it described the old docker-host runner (removed 2026-07-01, iac-repo#32). The dedicated runner VM mounts the DinD socket into every job container on purpose (`forgejo-runner-ansible`: `forgejo_runner_container_options: "-v /var/run/docker.sock:/var/run/docker.sock"` — *"so `docker build` works inside CI jobs"*). Comment corrected in `109c53a`; the new `image` job proves it in practice. - **Still open:** push mechanism (runner → host over SSH vs. a pull-based agent) and whether prod wants a manual approval gate or "tag = intent to ship". ### Recommended sequence 1. Fix iac-repo#159 (or establish it's environmental and bound it). 2. Dev auto-deploy first — `main` on green CI. Low blast radius, and it exercises the whole path. 3. Prod on `v*` tags once dev has run clean for a while. Not starting the implementation until (1) lands — building a deploy pipeline on a signal we currently can't trust would bake the flake into releases.

claude-bot referenced this issue

2026-07-17 03:39:23 +00:00

CI should build the production Docker image to catch Dockerfile/peer-dep regressions #103

claude-bot commented

2026-07-17 16:15:31 +00:00

Author

Contributor

Yes — Watchtower is the right call, and it's already your pattern

docker-host already runs per-stack Watchtower instances (marginalia-watchtower, chore-machine-watchtower, nickfedor/watchtower:latest), so this isn't new infrastructure — it's extending an established convention. That also settles the two open questions from my earlier comment:

Push mechanism → registry + Watchtower, not SSH-from-runner. Cleaner: CI builds and pushes an image; the host pulls a tag. No deploy key, no runner→host SSH.
Prod gate → tag = ship. The gate is "CI is green before the image is pushed" (tests + build must pass), and a version tag is the deliberate act. Watchtower redeploys prod when the watched tag's digest changes.

And it resolves the docker-socket tension honestly: I argued against an autoheal sidecar on #137 partly because it needs /var/run/docker.sock (root-equivalent). Watchtower needs the same socket — but you've already accepted that trade for two other stacks, and per-stack Watchtower instances keep the blast radius contained. For a routine, high-value workflow that's a defensible cost in a way a rare-brownout fix wasn't.

What it takes (the actual work, when you want it)

CI builds AND pushes to Forgejo's own registry (it's live — git.rhoving.com/v2/). #103 already builds both images; add a push step:
- on push to main → …/tealeaves-api:main (+ app) for dev auto-deploy
- on v* tag → …/tealeaves-api:release (moving tag) for prod
- pass GIT_SHA/GIT_DESCRIBE as build args so the admin version panel stays truthful (the stamp is baked into the image at build time).
Compose references the registry image on a moving tag (image: git.rhoving.com/rbrooks/tealeaves-api:release) instead of building locally, so there's a digest for Watchtower to watch. Prod and dev point at their respective tags.
A tealeaves-watchtower (mirroring the marginalia one) scoped to the TeaLeaves containers, watching that tag. Optionally trigger it from CI via Watchtower's HTTP API for an immediate deploy instead of waiting on its poll interval.

Migrations need no special handling — they run on API boot, so a pulled image self-migrates on recreate.

The one caveat to design around

Watchtower does not health-gate or roll back: it recreates the container and leans on the restart policy + healthcheck. A bad release still auto-deploys. Mitigations already in place: CI must be green before the image is pushed (so tsc + 261 tests + image build gate it), and #97's /health + depends_on: service_healthy stop nginx proxying to a not-yet-ready API. If you want true rollback-on-failure later, that's a Watchtower lifecycle-hook or a small deploy wrapper — worth a follow-up, not v1.

Prereq reminder: the CI flake that was blocking this is fixed (128e17b) with two consecutive green API runs since — so the "don't gate deploys on a flaky signal" objection is gone. Ready to build when you are; leaving this open pending your go-ahead on the CI-push + registry-image change.

## Yes — Watchtower is the right call, and it's already your pattern docker-host already runs per-stack Watchtower instances (`marginalia-watchtower`, `chore-machine-watchtower`, `nickfedor/watchtower:latest`), so this isn't new infrastructure — it's extending an established convention. That also settles the two open questions from my earlier comment: - **Push mechanism → registry + Watchtower**, not SSH-from-runner. Cleaner: CI builds and pushes an image; the host pulls a tag. No deploy key, no runner→host SSH. - **Prod gate → tag = ship.** The gate is "CI is green before the image is pushed" (tests + build must pass), and a version tag is the deliberate act. Watchtower redeploys prod when the watched tag's digest changes. And it resolves the docker-socket tension honestly: I argued against an autoheal sidecar on #137 partly because it needs `/var/run/docker.sock` (root-equivalent). Watchtower needs the same socket — **but you've already accepted that trade for two other stacks**, and per-stack Watchtower instances keep the blast radius contained. For a routine, high-value workflow that's a defensible cost in a way a rare-brownout fix wasn't. ## What it takes (the actual work, when you want it) 1. **CI builds AND pushes** to Forgejo's own registry (it's live — `git.rhoving.com/v2/`). #103 already builds both images; add a push step: - on push to `main` → `…/tealeaves-api:main` (+ app) for **dev auto-deploy** - on `v*` tag → `…/tealeaves-api:release` (moving tag) for **prod** - pass `GIT_SHA`/`GIT_DESCRIBE` as build args so the admin version panel stays truthful (the stamp is baked into the image at build time). 2. **Compose references the registry image** on a moving tag (`image: git.rhoving.com/rbrooks/tealeaves-api:release`) instead of building locally, so there's a digest for Watchtower to watch. Prod and dev point at their respective tags. 3. **A `tealeaves-watchtower`** (mirroring the marginalia one) scoped to the TeaLeaves containers, watching that tag. Optionally trigger it from CI via Watchtower's HTTP API for an immediate deploy instead of waiting on its poll interval. Migrations need no special handling — they run on API boot, so a pulled image self-migrates on recreate. ## The one caveat to design around Watchtower does **not** health-gate or roll back: it recreates the container and leans on the restart policy + healthcheck. A bad release still auto-deploys. Mitigations already in place: CI must be green before the image is pushed (so `tsc` + 261 tests + image build gate it), and #97's `/health` + `depends_on: service_healthy` stop nginx proxying to a not-yet-ready API. If you want true rollback-on-failure later, that's a Watchtower lifecycle-hook or a small deploy wrapper — worth a follow-up, not v1. **Prereq reminder:** the CI flake that was blocking this is fixed (`128e17b`) with two consecutive green API runs since — so the "don't gate deploys on a flaky signal" objection is gone. Ready to build when you are; leaving this open pending your go-ahead on the CI-push + registry-image change.

claude-bot removed this from the v7.2.0 - Reliability, data integrity & docs milestone

2026-07-17 16:34:33 +00:00