rbrooks/WeatherBot

Watch

Fork 0

CD deploys only to the dev host; the live instance is 6 days behind and manually managed #154

New issue

Closed

opened 2026-07-28 03:40:47 +00:00 by claude-bot · 3 comments

claude-bot commented

2026-07-28 03:40:47 +00:00

Contributor

CD works correctly. It just does not deploy the instance that is actually serving alerts. There are two separate WeatherBot deployments on two different Docker hosts, and only one of them receives merges.

Evidence

CD target (receives every merge). From the run 126 deploy job log:

cd ~/projects/WeatherBot
Deploying APP_VERSION=v1.7.0-25-g0ec6f84 (previous ref b6b22a26...)
 Container weatherbot-app-1  Recreated
 Container weatherbot-app-1  Started
curl ... http://localhost:18082/health
Deploy healthy:

The previous ref being b6b22a2 confirms the earlier merge landed there too. This side is healthy and current.

The live instance (never receives merges). Inspecting the running stack:

	value
compose config	`/home/ryan/WeatherBot/docker-compose.yml`
container `created`	2026-07-22 17:15:54
`StartedAt`	2026-07-22 17:15:57
`restart_count`	0
image `weatherbot-app:latest` built	2026-07-22 17:15:53
published port	`0.0.0.0:8054 -> 8000`

Untouched for six days and never restarted, across at least four successful CD runs on 2026-07-28 (00:58, 01:47, 02:13, 03:14).

They cannot be the same daemon. Both directories yield compose project name weatherbot, so a docker compose up from ~/projects/WeatherBot would have recreated these exact containers. It did not — they still show restart_count: 0 and a 07-22 creation time. Different hosts. The ports differ too: CD health-checks :18082, the live stack publishes :8054.

Behavioural confirmation. As of 03:36–03:39, after CD's 03:14 deploy, the live instance still logs the pre-#151 format (Webex all-clear failed ... Client error '400 Bad Request') with no Webex API rejected the message line and no Giving up on all-clear. It is running roughly v1.7.0.

Impact

The live instance is missing everything merged since 2026-07-22, including all five alert-correctness fixes:

#145 false all-clear on supersession — the bug that started this
#146 pagination truncation
#147 expiry sweep without a feed cross-check
#148 unbounded all-clear retry
#152 malformed all-clear image URL
plus #129 / #130 history capture and the dependency refresh

Live symptoms right now: Processing 2 pending lifted alert records every cycle, and notifier:webex failure_count=1238 climbing by roughly 2 per minute.

The real hazard

A green CD run currently implies a production deploy that is not happening. That assumption is easy to make — I made it earlier in this session — and it is most dangerous for exactly the class of fix involved here: an alert-correctness change can look shipped while the instance sending real notifications is unchanged.

Options

Add a production deploy job to cd.yml, gated on a PROD_SSH_KEY / PROD_HOST pair, mirroring the dev job (including its rollback-on-unhealthy behaviour).
Keep production manual but make it explicit — document that cd.yml targets dev only, and rename the job/workflow so "CD succeeded" cannot be read as "production is live".
Consolidate — if the dev instance is not earning its keep, point CD at production and drop the second stack.

Option 1 is the obvious one if production should track main; option 2 is right if production is deliberately pinned to releases. Either way the current state — where the two are silently different and nothing says so — is the thing to fix.

Immediate

Production needs a manual deploy to pick up the alert-correctness fixes:

cd /home/ryan/WeatherBot && git pull --ff-only origin main && docker compose up -d --build app

**CD works correctly. It just does not deploy the instance that is actually serving alerts.** There are two separate WeatherBot deployments on two different Docker hosts, and only one of them receives merges. ## Evidence **CD target (receives every merge).** From the run 126 deploy job log: ``` cd ~/projects/WeatherBot Deploying APP_VERSION=v1.7.0-25-g0ec6f84 (previous ref b6b22a26...) Container weatherbot-app-1 Recreated Container weatherbot-app-1 Started curl ... http://localhost:18082/health Deploy healthy: ``` The `previous ref` being `b6b22a2` confirms the earlier merge landed there too. This side is healthy and current. **The live instance (never receives merges).** Inspecting the running stack: | | value | |---|---| | compose config | `/home/ryan/WeatherBot/docker-compose.yml` | | container `created` | **2026-07-22 17:15:54** | | `StartedAt` | 2026-07-22 17:15:57 | | `restart_count` | **0** | | image `weatherbot-app:latest` built | 2026-07-22 17:15:53 | | published port | `0.0.0.0:8054 -> 8000` | Untouched for six days and never restarted, across at least four successful CD runs on 2026-07-28 (00:58, 01:47, 02:13, 03:14). **They cannot be the same daemon.** Both directories yield compose project name `weatherbot`, so a `docker compose up` from `~/projects/WeatherBot` would have recreated these exact containers. It did not — they still show `restart_count: 0` and a 07-22 creation time. Different hosts. The ports differ too: CD health-checks `:18082`, the live stack publishes `:8054`. **Behavioural confirmation.** As of 03:36–03:39, after CD's 03:14 deploy, the live instance still logs the pre-#151 format (`Webex all-clear failed ... Client error '400 Bad Request'`) with no `Webex API rejected the message` line and no `Giving up on all-clear`. It is running roughly v1.7.0. ## Impact The live instance is missing everything merged since 2026-07-22, including all five alert-correctness fixes: - #145 false all-clear on supersession — the bug that started this - #146 pagination truncation - #147 expiry sweep without a feed cross-check - #148 unbounded all-clear retry - #152 malformed all-clear image URL - plus #129 / #130 history capture and the dependency refresh Live symptoms right now: `Processing 2 pending lifted alert records` every cycle, and `notifier:webex failure_count=1238` climbing by roughly 2 per minute. ## The real hazard **A green CD run currently implies a production deploy that is not happening.** That assumption is easy to make — I made it earlier in this session — and it is most dangerous for exactly the class of fix involved here: an alert-correctness change can look shipped while the instance sending real notifications is unchanged. ## Options 1. **Add a production deploy job** to `cd.yml`, gated on a `PROD_SSH_KEY` / `PROD_HOST` pair, mirroring the dev job (including its rollback-on-unhealthy behaviour). 2. **Keep production manual but make it explicit** — document that `cd.yml` targets dev only, and rename the job/workflow so "CD succeeded" cannot be read as "production is live". 3. **Consolidate** — if the dev instance is not earning its keep, point CD at production and drop the second stack. Option 1 is the obvious one if production should track `main`; option 2 is right if production is deliberately pinned to releases. Either way the current state — where the two are silently different and nothing says so — is the thing to fix. ## Immediate Production needs a manual deploy to pick up the alert-correctness fixes: ``` cd /home/ryan/WeatherBot && git pull --ff-only origin main && docker compose up -d --build app ```

claude-bot added the

bug

type:infra

area:ci-cd

labels

2026-07-28 03:40:47 +00:00

claude-bot commented

2026-07-28 16:12:03 +00:00

Author

Contributor

Decision

Production updates only on stable release tags — not on every merge to main. Option 1 (auto-deploy on merge) is rejected; option 3 (consolidate) is rejected. main continues to deploy to dev, unchanged.

Mechanism: a Forgejo Actions job on v* tag push that invokes an Ansible playbook in iac-repo. Watchtower was considered and rejected for this app — see below.

Why not Watchtower

It was a reasonable thing to ask, since it is in use on other projects here. Two blockers, one structural and one safety:

There is no image to pull. docker-compose.yml uses build: with an APP_VERSION build arg, and the prod deploy is git pull + docker compose up -d --build. Watchtower updates containers by pulling a newer image for a tag from a registry, so it has nothing to act on until WeatherBot publishes images.
No health-gated rollback, and migrations run on startup. Alembic runs in the FastAPI lifespan, so a bad migration takes the app down with a half-migrated database and nothing watching. The existing dev job rolls back on unhealthy; giving that up on the instance that sends real weather warnings is the wrong trade.

What following the `iac-repo` convention actually implies

Worth stating plainly, because it is more than "add a playbook."

The established pattern for an app in iac-repo — carriage-ansible is the closest analogue — is registry-image based, not git-checkout based:

# carriage-ansible/roles/carriage/defaults/main.yml
# Pinned image — CI in the app repo builds and pushes this on a version tag.
# Never use :latest here; bump the tag to deploy a new version.
carriage_image: "git.rhoving.com/rbrooks/carriage:0.8.0"

Its role renders .env and docker-compose.yml from templates (secrets from Ansible Vault), does a docker_login against the Forgejo registry, pulls the pinned image, and brings the stack up with docker_compose_v2 under a service user in /opt/.

WeatherBot production today is none of those things: a hand-managed git checkout at /home/ryan/WeatherBot, running as ryan (uid 1000), with a hand-written .env, building images locally. Both live on the same docker-host that Ansible already manages for carriage, authentik, forgejo and shelf-life.

So converging on the convention means three pieces of work, not one:

Publish images. A CI job on v* tags that builds and pushes git.rhoving.com/rbrooks/weatherbot:<tag>. Bakes APP_VERSION at build time, which also removes the git checkout from prod entirely — and with it the root-owned-git-objects hazard that the current manual recipe has to work around.
weatherbot-ansible/ following the carriage layout: templated .env + docker-compose.yml, pinned image, health gate, rollback.
Migrate production onto it. This is the part that needs care, below.

⚠️ The migration hazard, stated before anyone starts

Compose derives its project name from the directory. Production currently runs as project weatherbot out of /home/ryan/WeatherBot, with its data in named volumes weatherbot_postgres_data, weatherbot_radar_cache, weatherbot_matrix_store, weatherbot_signal_cli_data.

Deploying the same stack from a new Ansible-managed path (say /opt/weatherbot-svc) without pinning the project name would create a brand-new parallel stack and orphan every one of those volumes — losing the alert history, the 155k freshness samples, and the history capture that only started on 2026-07-28. It would look like a successful deploy.

Mitigation is straightforward but must be explicit: pin COMPOSE_PROJECT_NAME=weatherbot in the rendered environment and keep the existing named volumes, or perform a deliberate, backed-up volume migration. Either way it is a decision to make up front, not something to discover afterwards.

Secrets also have to move. The prod .env holds the Postgres password, SECRET_KEY, the Authentik client secret, the Discord bot token, and the LIGHTNING_PLACEFILE_URL subscriber token. These would move into inventory/group_vars/weatherbot/vault.yml under ansible-vault, matching how carriage sources vault_carriage_secret_key.

Sequencing

Because of the above, the safe order is: (1) publish images on tag → (2) write and dry-run the playbook against the dev host → (3) cut production over, with a database backup taken first. Dev is the natural rehearsal for the cutover, and it already receives every merge.

Interim state

Until this lands, production remains manual. Today it was deployed by hand to 12d47dd to pick up #131 and #132. Anyone reading a green CD run should still not conclude production is current — that is precisely the hazard this issue exists to remove, and it is not removed yet.

## Decision **Production updates only on stable release tags** — not on every merge to `main`. Option 1 (auto-deploy on merge) is rejected; option 3 (consolidate) is rejected. `main` continues to deploy to dev, unchanged. **Mechanism: a Forgejo Actions job on `v*` tag push that invokes an Ansible playbook in `iac-repo`.** Watchtower was considered and rejected for this app — see below. ## Why not Watchtower It was a reasonable thing to ask, since it is in use on other projects here. Two blockers, one structural and one safety: 1. **There is no image to pull.** `docker-compose.yml` uses `build:` with an `APP_VERSION` build arg, and the prod deploy is `git pull` + `docker compose up -d --build`. Watchtower updates containers by pulling a newer image for a tag from a registry, so it has nothing to act on until WeatherBot publishes images. 2. **No health-gated rollback, and migrations run on startup.** Alembic runs in the FastAPI lifespan, so a bad migration takes the app down with a half-migrated database and nothing watching. The existing dev job rolls back on unhealthy; giving that up on the instance that sends real weather warnings is the wrong trade. ## What following the `iac-repo` convention actually implies Worth stating plainly, because it is more than "add a playbook." The established pattern for an app in `iac-repo` — `carriage-ansible` is the closest analogue — is **registry-image based, not git-checkout based**: ```yaml # carriage-ansible/roles/carriage/defaults/main.yml # Pinned image — CI in the app repo builds and pushes this on a version tag. # Never use :latest here; bump the tag to deploy a new version. carriage_image: "git.rhoving.com/rbrooks/carriage:0.8.0" ``` Its role renders `.env` and `docker-compose.yml` from templates (secrets from Ansible Vault), does a `docker_login` against the Forgejo registry, pulls the **pinned** image, and brings the stack up with `docker_compose_v2` under a service user in `/opt/`. WeatherBot production today is none of those things: a hand-managed git checkout at `/home/ryan/WeatherBot`, running as `ryan` (uid 1000), with a hand-written `.env`, building images locally. Both live on the same `docker-host` that Ansible already manages for carriage, authentik, forgejo and shelf-life. So converging on the convention means three pieces of work, not one: 1. **Publish images.** A CI job on `v*` tags that builds and pushes `git.rhoving.com/rbrooks/weatherbot:<tag>`. Bakes `APP_VERSION` at build time, which also removes the git checkout from prod entirely — and with it the root-owned-git-objects hazard that the current manual recipe has to work around. 2. **`weatherbot-ansible/`** following the carriage layout: templated `.env` + `docker-compose.yml`, pinned image, health gate, rollback. 3. **Migrate production onto it.** This is the part that needs care, below. ## ⚠️ The migration hazard, stated before anyone starts **Compose derives its project name from the directory.** Production currently runs as project `weatherbot` out of `/home/ryan/WeatherBot`, with its data in named volumes `weatherbot_postgres_data`, `weatherbot_radar_cache`, `weatherbot_matrix_store`, `weatherbot_signal_cli_data`. Deploying the same stack from a new Ansible-managed path (say `/opt/weatherbot-svc`) without pinning the project name would create a **brand-new parallel stack** and orphan every one of those volumes — losing the alert history, the 155k freshness samples, and the history capture that only started on 2026-07-28. It would look like a successful deploy. Mitigation is straightforward but must be explicit: pin `COMPOSE_PROJECT_NAME=weatherbot` in the rendered environment and keep the existing named volumes, or perform a deliberate, backed-up volume migration. Either way it is a decision to make up front, not something to discover afterwards. **Secrets also have to move.** The prod `.env` holds the Postgres password, `SECRET_KEY`, the Authentik client secret, the Discord bot token, and the `LIGHTNING_PLACEFILE_URL` subscriber token. These would move into `inventory/group_vars/weatherbot/vault.yml` under `ansible-vault`, matching how carriage sources `vault_carriage_secret_key`. ## Sequencing Because of the above, the safe order is: **(1) publish images on tag → (2) write and dry-run the playbook against the dev host → (3) cut production over, with a database backup taken first.** Dev is the natural rehearsal for the cutover, and it already receives every merge. ## Interim state Until this lands, production remains **manual**. Today it was deployed by hand to `12d47dd` to pick up #131 and #132. Anyone reading a green CD run should still not conclude production is current — that is precisely the hazard this issue exists to remove, and it is not removed yet.

rbrooks referenced this issue from a commit

2026-07-28 16:40:28 +00:00

Deploy production on release tags, not on merge (#154)

claude-bot referenced this issue from a pull request that will close it,

2026-07-28 16:40:49 +00:00

Publish release images on v* tags; production deploys from iac-repo (#154) #159

claude-bot commented

2026-07-28 16:41:26 +00:00

Author

Contributor

Both PRs are up

Repo	PR	Contents	CI
`Rhoving/iac-repo`	#220	`weatherbot-ansible/` role, inventory group, vault stub	✅ green (2m59s)
`rbrooks/WeatherBot`	#159	`release.yml` (build+push+deploy on `v*`), `cd.yml` renamed	✅ green (5m00s)

Tracked on the iac side as Rhoving/iac-repo#219. Merge order matters: iac-repo#220 first — the workflow invokes a play that does not exist until it lands.

What landed against each option on this issue

Option 1 (deploy on merge) and option 3 (consolidate) are rejected. What is implemented is closest to option 2 — production stays deliberate — but automated rather than manual: a v* tag is the trigger, so production can never silently lag the way it did here, and nobody has to remember a procedure.

cd.yml is also renamed "CD (dev only)" with a header saying so, which was the explicit ask in option 2. The hazard was never that CD was broken, only that its name implied more than it did.

Two things deliberately left for a human

The production cutover. The play is written but not run. Production is a git checkout at /home/ryan/WeatherBot running as uid 1000 with hand-written secrets; moving it to /opt/weatherbot-svc under a service user touches the four named volumes holding all the alert history. The compose project name is pinned to weatherbot in the rendered file specifically so those volumes survive, but the cutover itself should be watched, after a pg_dump and a dev rehearsal. Sequence is on iac-repo#220.

One open design question, on WeatherBot#159: the deploy passes -e weatherbot_image=<tag>, so the committed pin in iac-repo shows the previous version until someone bumps it. That is convenient but leaves git and reality disagreeing, which cuts against how the rest of the fleet works. The alternative — fail the play when the pin and the tag disagree — keeps git authoritative at the cost of a two-repo ordering requirement. Worth a decision before this gets used in anger.

Incidentally fixed

Production had no database backup of any kind. Upstream ships an opt-in db-backup compose profile, but it bind-mounts ./scripts from a git checkout — unusable from a registry image — and was never enabled. iac-repo#220 adds a nightly pg_dump systemd timer instead. That matters more than usual here: the play's rollback restores the previous image, not the previous schema, and migrations are forward-only, so a bad migration needs a restore to recover from. Until now there would have been nothing to restore from.

## Both PRs are up | Repo | PR | Contents | CI | |---|---|---|---| | `Rhoving/iac-repo` | [#220](https://git.rhoving.com/Rhoving/iac-repo/pulls/220) | `weatherbot-ansible/` role, inventory group, vault stub | ✅ green (2m59s) | | `rbrooks/WeatherBot` | [#159](https://git.rhoving.com/rbrooks/WeatherBot/pulls/159) | `release.yml` (build+push+deploy on `v*`), `cd.yml` renamed | ✅ green (5m00s) | Tracked on the iac side as [Rhoving/iac-repo#219](https://git.rhoving.com/Rhoving/iac-repo/issues/219). **Merge order matters: iac-repo#220 first** — the workflow invokes a play that does not exist until it lands. ## What landed against each option on this issue Option 1 (deploy on merge) and option 3 (consolidate) are rejected. What is implemented is closest to option 2 — production stays deliberate — but automated rather than manual: a `v*` tag is the trigger, so production can never silently lag the way it did here, and nobody has to remember a procedure. `cd.yml` is also renamed **"CD (dev only)"** with a header saying so, which was the explicit ask in option 2. The hazard was never that CD was broken, only that its name implied more than it did. ## Two things deliberately left for a human **The production cutover.** The play is written but not run. Production is a git checkout at `/home/ryan/WeatherBot` running as uid 1000 with hand-written secrets; moving it to `/opt/weatherbot-svc` under a service user touches the four named volumes holding all the alert history. The compose project name is pinned to `weatherbot` in the rendered file specifically so those volumes survive, but the cutover itself should be watched, after a `pg_dump` and a dev rehearsal. Sequence is on iac-repo#220. **One open design question**, on WeatherBot#159: the deploy passes `-e weatherbot_image=<tag>`, so the committed pin in iac-repo shows the *previous* version until someone bumps it. That is convenient but leaves git and reality disagreeing, which cuts against how the rest of the fleet works. The alternative — fail the play when the pin and the tag disagree — keeps git authoritative at the cost of a two-repo ordering requirement. Worth a decision before this gets used in anger. ## Incidentally fixed Production had **no database backup of any kind**. Upstream ships an opt-in `db-backup` compose profile, but it bind-mounts `./scripts` from a git checkout — unusable from a registry image — and was never enabled. iac-repo#220 adds a nightly `pg_dump` systemd timer instead. That matters more than usual here: the play's rollback restores the previous *image*, not the previous *schema*, and migrations are forward-only, so a bad migration needs a restore to recover from. Until now there would have been nothing to restore from.

claude-bot commented

2026-08-01 00:12:57 +00:00

Author

Contributor

Decision revised: CI publishes, iac-repo deploys

Ryan's call, 2026-07-31: the release workflow should only build and push the image. Deployment happens from iac-repo by an operator running the play. Rationale:

A deploy-from-CI job needs PROD_SSH_KEY + ANSIBLE_VAULT_PASSWORD in Forgejo Actions secrets — together root-equivalent on docker-host, exposed to anything that can push a v* tag or compromise a runner. Publish-only leaves CI with a package:write token, whose worst case (a poisoned image) still has to get past the operator.
This resolves the open design question from the previous comment (the -e weatherbot_image=<tag> pin drift): with deploys operator-run, the committed weatherbot_image pin in iac-repo is the deployed state. Git stays authoritative, matching the fleet convention.

PR #159 updated accordingly (commit 42ae241): deploy-prod job removed, secrets requirement reduced to REGISTRY_TOKEN only, README documents the three-step release procedure (tag → bump pin → run play). iac-repo#220 is unchanged by this — the role was already invocation-agnostic — and the merge-order constraint between the two PRs is gone.

Trade-off accepted knowingly: shipping regains one human step (bump pin + run play), which is the same class of step whose forgetability caused this issue. Mitigations: the tag itself is already a deliberate human act (so the deploy step is adjacent, not separate), the release run emits a ::notice:: with the exact commands, and the committed pin makes any lag visible — git log on iac-repo answers "what is production running", which was the unanswerable question that started this.

## Decision revised: CI publishes, iac-repo deploys Ryan's call, 2026-07-31: the release workflow should **only build and push the image**. Deployment happens from iac-repo by an operator running the play. Rationale: - A deploy-from-CI job needs `PROD_SSH_KEY` + `ANSIBLE_VAULT_PASSWORD` in Forgejo Actions secrets — together root-equivalent on `docker-host`, exposed to anything that can push a `v*` tag or compromise a runner. Publish-only leaves CI with a `package:write` token, whose worst case (a poisoned image) still has to get past the operator. - This resolves the open design question from the previous comment (the `-e weatherbot_image=<tag>` pin drift): with deploys operator-run, the committed `weatherbot_image` pin in iac-repo **is** the deployed state. Git stays authoritative, matching the fleet convention. PR #159 updated accordingly (commit 42ae241): `deploy-prod` job removed, secrets requirement reduced to `REGISTRY_TOKEN` only, README documents the three-step release procedure (tag → bump pin → run play). iac-repo#220 is unchanged by this — the role was already invocation-agnostic — and the merge-order constraint between the two PRs is gone. Trade-off accepted knowingly: shipping regains one human step (bump pin + run play), which is the same class of step whose forgetability caused this issue. Mitigations: the tag itself is already a deliberate human act (so the deploy step is adjacent, not separate), the release run emits a `::notice::` with the exact commands, and the committed pin makes any lag *visible* — `git log` on iac-repo answers "what is production running", which was the unanswerable question that started this.

claude-bot closed this issue

2026-08-01 00:22:22 +00:00

claude-bot referenced this issue from a commit

2026-08-01 00:22:24 +00:00

Merge pull request 'Publish release images on v* tags; production deploys from iac-repo (#154)' (#159) from feat/release-image-publish into main

rbrooks referenced this issue from a commit

2026-08-01 01:10:06 +00:00

chore(ci): retire the merge-time dev deploy — both hosts are Ansible-managed now

claude-bot referenced this issue

2026-08-01 01:10:17 +00:00

chore(ci): retire the merge-time dev deploy — both hosts are Ansible-managed now #160