rbrooks/WeatherBot

Watch

Fork 0

INCIDENT: production has delivered no alerts since 2026-08-11 — crash-looping on a stale image against a forward-migrated DB #186

New issue

Closed

opened 2026-08-20 19:17:53 +00:00 by claude-bot · 4 comments

claude-bot commented

2026-08-20 19:17:53 +00:00

Collaborator

Severity: high. Production severe-weather alerting has been silently down for ~9.5 days. Found incidentally on 2026-08-20 while querying production for an unrelated radar_animated distribution (#184).

Impact

Last alert delivered: 2026-08-11 03:27:05 UTC. Nothing since. sent_alerts holds 1,719 rows and has not grown in 9.5 days.
The app container has restarted 13,294 times and is still restarting.
Nothing detected or reported this. The stack's other containers (postgres, signal-cli) are Up 9 days (healthy), so a container-level health view looks partly fine.

Root cause

The database is at Alembic revision 0033 (shipped in v1.8.1). The container is running weatherbot-app — a locally-built image reporting WEATHERBOT_VERSION=v1.7.0-29-g12d47dd, which predates 0033 and does not contain it. Startup therefore dies every time:

ERROR [alembic.util.messaging] Can't locate revision identified by '0033'
FAILED: Can't locate revision identified by '0033'
subprocess.CalledProcessError: Command '[... '-m', 'alembic', 'upgrade', 'head']' returned non-zero exit status 255
ERROR:    Application startup failed. Exiting.

app/main.py's lifespan runs alembic upgrade head with check=True, so a migration failure is fatal to startup by design — correct behaviour, but it means a code/DB mismatch is an unrecoverable crash loop rather than a degraded start.

How the mismatch happened

The running container's compose labels point at the old pre-Ansible checkout:

com.docker.compose.project             = weatherbot
com.docker.compose.project.config_files = /home/ryan/WeatherBot/docker-compose.yml
com.docker.compose.project.working_dir  = /home/ryan/WeatherBot

while the Ansible-managed deployment lives at /opt/weatherbot-svc. Both use the compose project name weatherbot, so they own the same containers. Something ran docker compose up from /home/ryan/WeatherBot on ~2026-08-11 05:54 (container Created timestamp), which rebuilt the app from that stale checkout and replaced the Ansible-deployed v1.8.1 container — while leaving the database forward-migrated at 0033.

This is exactly the failure mode .forgejo/workflows/cd.yml documents as the reason the merge-time dev deploy was retired: "it rebuilt the stack from a git checkout under the same compose project name the Ansible role owns." The same landmine existed on production and was never disarmed.

weatherbot.service (the Ansible-managed systemd unit) is present but in failed state.

Proposed remediation

Restore service — run the weatherbot play against docker-host. The group_vars pin is 1.8.1, whose schema is exactly 0033, so it matches the database with no migration needed. This is the committed intended state.
Disarm the landmine — /home/ryan/WeatherBot must stop being able to clobber the Ansible stack. Either remove it, or rename its compose project so it can never again own weatherbot's containers. Restoring service without this leaves the same trap armed.
Detection gap — 9.5 days of total outage with no alarm is the more serious finding than the outage itself. The poller heartbeat (I-07) detects upstream staleness, but nothing noticed the app was not running at all. Needs its own issue.

Not the cause

The weatherbot-backup.timer is active and its service inactive dead (normal between runs). Postgres and signal-cli are healthy. This is purely the app container.

Incidental finding (the query that surfaced this)

notification_channels on production: 4 rows, all radar_animated = true. So #184's migration 0037 backfill is a no-op on production as well as dev — no channel has opted out of radar, so none can be wrongly switched on. The migration remains correct; there is simply no row that exercises it.

**Severity: high. Production severe-weather alerting has been silently down for ~9.5 days.** Found incidentally on 2026-08-20 while querying production for an unrelated `radar_animated` distribution (#184). ## Impact - **Last alert delivered: `2026-08-11 03:27:05 UTC`.** Nothing since. `sent_alerts` holds 1,719 rows and has not grown in 9.5 days. - The app container has restarted **13,294 times** and is still restarting. - Nothing detected or reported this. The stack's other containers (`postgres`, `signal-cli`) are `Up 9 days (healthy)`, so a container-level health view looks partly fine. ## Root cause The database is at Alembic revision **`0033`** (shipped in v1.8.1). The container is running **`weatherbot-app`** — a locally-built image reporting `WEATHERBOT_VERSION=v1.7.0-29-g12d47dd`, which predates `0033` and does not contain it. Startup therefore dies every time: ``` ERROR [alembic.util.messaging] Can't locate revision identified by '0033' FAILED: Can't locate revision identified by '0033' subprocess.CalledProcessError: Command '[... '-m', 'alembic', 'upgrade', 'head']' returned non-zero exit status 255 ERROR: Application startup failed. Exiting. ``` `app/main.py`'s lifespan runs `alembic upgrade head` with `check=True`, so a migration failure is fatal to startup by design — correct behaviour, but it means a code/DB mismatch is an unrecoverable crash loop rather than a degraded start. ### How the mismatch happened The running container's compose labels point at the **old pre-Ansible checkout**: ``` com.docker.compose.project = weatherbot com.docker.compose.project.config_files = /home/ryan/WeatherBot/docker-compose.yml com.docker.compose.project.working_dir = /home/ryan/WeatherBot ``` while the Ansible-managed deployment lives at `/opt/weatherbot-svc`. **Both use the compose project name `weatherbot`**, so they own the same containers. Something ran `docker compose up` from `/home/ryan/WeatherBot` on ~2026-08-11 05:54 (container `Created` timestamp), which rebuilt the app from that stale checkout and replaced the Ansible-deployed v1.8.1 container — while leaving the database forward-migrated at `0033`. This is exactly the failure mode `.forgejo/workflows/cd.yml` documents as the reason the merge-time dev deploy was retired: *"it rebuilt the stack from a git checkout under the same compose project name the Ansible role owns."* The same landmine existed on production and was never disarmed. `weatherbot.service` (the Ansible-managed systemd unit) is present but in **`failed`** state. ## Proposed remediation 1. **Restore service** — run the weatherbot play against `docker-host`. The `group_vars` pin is `1.8.1`, whose schema is exactly `0033`, so it matches the database with no migration needed. This is the committed intended state. 2. **Disarm the landmine** — `/home/ryan/WeatherBot` must stop being able to clobber the Ansible stack. Either remove it, or rename its compose project so it can never again own `weatherbot`'s containers. Restoring service without this leaves the same trap armed. 3. **Detection gap** — 9.5 days of total outage with no alarm is the more serious finding than the outage itself. The poller heartbeat (I-07) detects *upstream* staleness, but nothing noticed the app was not running at all. Needs its own issue. ## Not the cause The `weatherbot-backup.timer` is `active` and its service `inactive dead` (normal between runs). Postgres and signal-cli are healthy. This is purely the app container. ## Incidental finding (the query that surfaced this) `notification_channels` on production: **4 rows, all `radar_animated = true`**. So #184's migration `0037` backfill is a no-op on production as well as dev — no channel has opted out of radar, so none can be wrongly switched on. The migration remains correct; there is simply no row that exercises it.

claude-bot commented

2026-08-20 22:03:18 +00:00

Author

Collaborator

Resolved — service restored 2026-08-20 ~21:47 UTC

Production is running git.rhoving.com/rbrooks/weatherbot:1.8.1 (the registry image the pin specifies, not a local build), healthy, with the full Ansible-managed stack recreated (postgres:16.14-alpine, signal-cli).

Confirmed working end-to-end, not merely container-healthy: all APScheduler jobs execute successfully (poll_alerts_job, poll_spc_job, process_delivery_outbox_job, …), and sent_alerts recorded a delivery at 2026-08-20 21:51:25 UTC — the first since 2026-08-11 03:27. Total outage: ~9.7 days. Database remains at 0033, matching 1.8.1, so no migration was needed.

Correcting the root cause above

My original attribution — "something ran docker compose up from /home/ryan/WeatherBot" — was stated with more confidence than the evidence supported. The journal now shows a more specific first domino:

Aug 11 05:39:38  Container weatherbot-db-1 Starting
Aug 11 05:40:44  Container weatherbot-db-1 Started          <- 66s to start
Aug 11 05:40:51  Container weatherbot-db-1 Waiting          <- waiting on health
Aug 11 05:40:59  weatherbot.service: start operation timed out. Terminating.
Aug 11 05:40:59  Main process exited, code=exited, status=130   (SIGINT)
Aug 11 05:40:59  Failed with result 'timeout'

weatherbot.service timed out during docker compose up because Postgres took 66s to become healthy, and systemd SIGINT'd compose with the stack half-started — the app container never came up under the unit.

This is precisely the failure the systemd-unit change in iac-repo#277 was written to fix, whose own comment reads: "is slow to become healthy, compose up stops waiting, and the unit dies with the stack half-started." Production had not received that change.

What happened between 05:40:59 (unit failed) and 05:54:48 (the crash-looping weatherbot-app container's creation timestamp) is not established by evidence I have. The 14-minute gap and the container's build-from-source origin at /home/ryan/WeatherBot are consistent with a manual recovery attempt from the old pre-Ansible checkout, but I did not find a log proving that, and I should not have asserted it as fact. What is certain is that the resulting container was built from stale v1.7.0-era source and could not satisfy the database's 0033, producing 13,294 restarts.

So the chain was: iac-repo#277 missing on prod → compose-up timeout on slow Postgres → half-started stack → recovery from a stale checkout → permanent version/schema mismatch.

Still outstanding

weatherbot.service remains in failed state (stale, from Aug 11) while the containers run healthy outside it. The unit file is now correct — WorkingDirectory=/opt/weatherbot-svc, TimeoutStartSec=300, and #277's Restart=on-failure / RestartSec=30 are all present — so this is leftover state, not a live fault. It wants systemctl reset-failed weatherbot.service followed by systemctl start weatherbot.service so the unit adopts the running stack. Until then, anything monitoring unit state reports a failure that no longer exists.
The landmine is still armed. /home/ryan/WeatherBot/docker-compose.yml still exists on docker-host, and weatherbot-app:latest is still in the local image store. Both should go, or the checkout's compose project renamed, so it can never again own the weatherbot containers.
The detection gap is the most important finding. ~9.7 days of complete outage on a life-safety service with nothing raising an alarm. The poller heartbeat detects upstream staleness; nothing detects "the app is not running." This deserves its own issue and should arguably gate v2.0.0.

## Resolved — service restored 2026-08-20 ~21:47 UTC Production is running `git.rhoving.com/rbrooks/weatherbot:1.8.1` (the registry image the pin specifies, not a local build), healthy, with the full Ansible-managed stack recreated (`postgres:16.14-alpine`, signal-cli). **Confirmed working end-to-end, not merely container-healthy:** all APScheduler jobs execute successfully (`poll_alerts_job`, `poll_spc_job`, `process_delivery_outbox_job`, …), and **`sent_alerts` recorded a delivery at `2026-08-20 21:51:25 UTC`** — the first since 2026-08-11 03:27. Total outage: **~9.7 days**. Database remains at `0033`, matching 1.8.1, so no migration was needed. ## Correcting the root cause above My original attribution — "something ran `docker compose up` from `/home/ryan/WeatherBot`" — was stated with more confidence than the evidence supported. The journal now shows a more specific first domino: ``` Aug 11 05:39:38 Container weatherbot-db-1 Starting Aug 11 05:40:44 Container weatherbot-db-1 Started <- 66s to start Aug 11 05:40:51 Container weatherbot-db-1 Waiting <- waiting on health Aug 11 05:40:59 weatherbot.service: start operation timed out. Terminating. Aug 11 05:40:59 Main process exited, code=exited, status=130 (SIGINT) Aug 11 05:40:59 Failed with result 'timeout' ``` **`weatherbot.service` timed out during `docker compose up` because Postgres took 66s to become healthy**, and systemd SIGINT'd compose with the stack half-started — the app container never came up under the unit. This is precisely the failure the systemd-unit change in **iac-repo#277** was written to fix, whose own comment reads: *"is slow to become healthy, `compose up` stops waiting, and the unit dies with the stack half-started."* Production had not received that change. What happened between `05:40:59` (unit failed) and `05:54:48` (the crash-looping `weatherbot-app` container's creation timestamp) is **not established by evidence I have**. The 14-minute gap and the container's build-from-source origin at `/home/ryan/WeatherBot` are consistent with a manual recovery attempt from the old pre-Ansible checkout, but I did not find a log proving that, and I should not have asserted it as fact. What is certain is that the resulting container was built from stale v1.7.0-era source and could not satisfy the database's `0033`, producing 13,294 restarts. So the chain was: **iac-repo#277 missing on prod → compose-up timeout on slow Postgres → half-started stack → recovery from a stale checkout → permanent version/schema mismatch.** ## Still outstanding 1. **`weatherbot.service` remains in `failed` state** (stale, from Aug 11) while the containers run healthy outside it. The unit file is now correct — `WorkingDirectory=/opt/weatherbot-svc`, `TimeoutStartSec=300`, and #277's `Restart=on-failure` / `RestartSec=30` are all present — so this is leftover state, not a live fault. It wants `systemctl reset-failed weatherbot.service` followed by `systemctl start weatherbot.service` so the unit adopts the running stack. Until then, anything monitoring unit state reports a failure that no longer exists. 2. **The landmine is still armed.** `/home/ryan/WeatherBot/docker-compose.yml` still exists on `docker-host`, and `weatherbot-app:latest` is still in the local image store. Both should go, or the checkout's compose project renamed, so it can never again own the `weatherbot` containers. 3. **The detection gap is the most important finding.** ~9.7 days of complete outage on a life-safety service with nothing raising an alarm. The poller heartbeat detects upstream staleness; nothing detects "the app is not running." This deserves its own issue and should arguably gate v2.0.0.

claude-bot referenced this issue

2026-08-21 03:43:35 +00:00

Notifier radar: make "animated" actually animate, and split the flag that doesn't mean what it says #184

claude-bot referenced this issue

2026-08-21 03:50:32 +00:00

Historical Weather Event Explorer #22

claude-bot commented

2026-10-13 15:21:33 +00:00

Author

Collaborator

Follow-up check on production, 2026-10-13 (read-only)

Status of the three items left outstanding on 2026-08-20:

#	Item	State now
1	`weatherbot.service` stuck in `failed`	Resolved. `active (exited)`, `is-failed` = `active`. Unit carries `WorkingDirectory=/opt/weatherbot-svc`, `TimeoutStartSec=300`, `Restart=on-failure`, `RestartSec=30`.
2	Stale checkout able to clobber the stack	Mostly disarmed. `/home/ryan/WeatherBot` still exists, but it has no `docker-compose.yml` (only `docker-compose.yml.retired` and a `.bak`), so a bare `docker compose up` there no longer does anything. The locally built `weatherbot-app:latest` image is still in the image store.
3	Nothing detects "the app is not running"	Still open, and still unfiled.

Running state: weatherbot:1.8.1 from /opt/weatherbot-svc, all three containers healthy, database at 0033, 5 locations, 4 channels.

The unit now recovers on its own

docker-host was down from 2026-10-11 00:55 to 06:08 UTC (clean stop, then a new boot). The stack came back through the systemd unit with no intervention: Postgres healthy in 5 seconds, app started at 06:09:16. That is the path that failed on 2026-08-11.

A three-week silence that is not an outage

sent_alerts has nothing between 2026-09-21 01:36 and 2026-10-13 05:34, which looked like a repeat. It is not:

product_freshness_samples shows the NWS alert poll succeeding 1,440 times a day, every day from 09-30 through 10-12, apart from 10-11 (1,125, matching the 5h13m host outage). Earlier days are past that table's 14-day retention.
lightning_clusters has rows on 09-22, 09-27 and 10-09, so the app was writing during the part the freshness table no longer covers.

So the gap is quiet weather. But telling the two apart took four queries against production, which is item 3 in practice: the same picture, with the app dead, would have looked identical from sent_alerts.

Not checked

Backup files. weatherbot-backup.timer is active and the last run (10-13 03:34 UTC) exited 0, but /opt/weatherbot-svc/backups is not readable by the login user, so the dumps themselves were not listed.

Left to do

Remove weatherbot-app:latest from docker-host (and optionally the leftover 2.0.0-rc3, 1.8.0, 1.7.1 images; rc3 is the candidate that must not be deployed).
Remove or archive /home/ryan/WeatherBot.
File the detection-gap issue.

## Follow-up check on production, 2026-10-13 (read-only) Status of the three items left outstanding on 2026-08-20: | # | Item | State now | |---|---|---| | 1 | `weatherbot.service` stuck in `failed` | **Resolved.** `active (exited)`, `is-failed` = `active`. Unit carries `WorkingDirectory=/opt/weatherbot-svc`, `TimeoutStartSec=300`, `Restart=on-failure`, `RestartSec=30`. | | 2 | Stale checkout able to clobber the stack | **Mostly disarmed.** `/home/ryan/WeatherBot` still exists, but it has no `docker-compose.yml` (only `docker-compose.yml.retired` and a `.bak`), so a bare `docker compose up` there no longer does anything. The locally built `weatherbot-app:latest` image is **still in the image store**. | | 3 | Nothing detects "the app is not running" | **Still open, and still unfiled.** | Running state: `weatherbot:1.8.1` from `/opt/weatherbot-svc`, all three containers healthy, database at `0033`, 5 locations, 4 channels. ## The unit now recovers on its own `docker-host` was down from **2026-10-11 00:55 to 06:08 UTC** (clean stop, then a new boot). The stack came back through the systemd unit with no intervention: Postgres healthy in 5 seconds, app started at 06:09:16. That is the path that failed on 2026-08-11. ## A three-week silence that is not an outage `sent_alerts` has nothing between 2026-09-21 01:36 and 2026-10-13 05:34, which looked like a repeat. It is not: - `product_freshness_samples` shows the NWS alert poll succeeding **1,440 times a day, every day** from 09-30 through 10-12, apart from 10-11 (1,125, matching the 5h13m host outage). Earlier days are past that table's 14-day retention. - `lightning_clusters` has rows on 09-22, 09-27 and 10-09, so the app was writing during the part the freshness table no longer covers. So the gap is quiet weather. But telling the two apart took four queries against production, which is item 3 in practice: the same picture, with the app dead, would have looked identical from `sent_alerts`. ## Not checked Backup files. `weatherbot-backup.timer` is active and the last run (10-13 03:34 UTC) exited 0, but `/opt/weatherbot-svc/backups` is not readable by the login user, so the dumps themselves were not listed. ## Left to do - [ ] Remove `weatherbot-app:latest` from `docker-host` (and optionally the leftover `2.0.0-rc3`, `1.8.0`, `1.7.1` images; rc3 is the candidate that must not be deployed). - [ ] Remove or archive `/home/ryan/WeatherBot`. - [ ] File the detection-gap issue.

claude-bot commented

2026-10-13 15:56:45 +00:00

Author

Collaborator

Detection gap (item 3) is now tracked as Rhoving/iac-repo#636. Per @rbrooks on 2026-10-13 it does not gate v2.0.0.

What remains on this issue is the production cleanup: the weatherbot-app:latest image and the /home/ryan/WeatherBot checkout.

Detection gap (item 3) is now tracked as Rhoving/iac-repo#636. Per @rbrooks on 2026-10-13 it does **not** gate v2.0.0. What remains on this issue is the production cleanup: the `weatherbot-app:latest` image and the `/home/ryan/WeatherBot` checkout.

claude-bot commented

2026-10-13 16:10:31 +00:00

Author

Collaborator

Production cleanup done, 2026-10-13 16:10 UTC — closing

Requested by @rbrooks. Run through Ansible from the control node; the running stack was not touched.

Old checkout removed. /home/ryan/WeatherBot is gone. Checked first:

HEAD was detached at 12d47dd, a merge commit already on main. No unpushed commits, no stashes; the only tracked change was the already-deleted docker-compose.yml.
Nothing under /etc/systemd/system and no crontab entry referenced the path.
Four files there existed nowhere else: .env, .env.bak.pre-v160-modernize, and two retired compose files. These were archived before deletion to /home/ryan/weatherbot-old-checkout-env-20261013.tar.gz on docker-host (mode 600, 3.2 KB). It holds the pre-Ansible secrets, so delete it once you are sure nothing in it is missing from the vault.

Unused images removed (none had a container; removed without force): weatherbot-app:latest, weatherbot:2.0.0-rc3, weatherbot:1.8.0, weatherbot:1.7.1. The three registry images can be pulled again if ever needed. Only weatherbot:1.8.1 remains.

After: all three containers Up 2 days (healthy) on 1.8.1, weatherbot.service active.

Where each original item ended up

Unit stuck in failed: resolved, and it recovered the stack unaided after the 2026-10-11 reboot.
Checkout and locally built image able to take over the stack: both removed today.
Nothing detects the app being down: tracked as Rhoving/iac-repo#636, not gating v2.0.0.

## Production cleanup done, 2026-10-13 16:10 UTC — closing Requested by @rbrooks. Run through Ansible from the control node; the running stack was not touched. **Old checkout removed.** `/home/ryan/WeatherBot` is gone. Checked first: - HEAD was detached at `12d47dd`, a merge commit already on `main`. No unpushed commits, no stashes; the only tracked change was the already-deleted `docker-compose.yml`. - Nothing under `/etc/systemd/system` and no crontab entry referenced the path. - Four files there existed nowhere else: `.env`, `.env.bak.pre-v160-modernize`, and two retired compose files. These were archived before deletion to `/home/ryan/weatherbot-old-checkout-env-20261013.tar.gz` on `docker-host` (mode 600, 3.2 KB). It holds the pre-Ansible secrets, so delete it once you are sure nothing in it is missing from the vault. **Unused images removed** (none had a container; removed without force): `weatherbot-app:latest`, `weatherbot:2.0.0-rc3`, `weatherbot:1.8.0`, `weatherbot:1.7.1`. The three registry images can be pulled again if ever needed. Only `weatherbot:1.8.1` remains. **After:** all three containers `Up 2 days (healthy)` on 1.8.1, `weatherbot.service` active. ## Where each original item ended up 1. Unit stuck in `failed`: resolved, and it recovered the stack unaided after the 2026-10-11 reboot. 2. Checkout and locally built image able to take over the stack: both removed today. 3. Nothing detects the app being down: tracked as Rhoving/iac-repo#636, not gating v2.0.0.

claude-bot closed this issue

2026-10-13 16:10:36 +00:00

claude-bot referenced this issue

2026-10-13 16:26:51 +00:00

Historical Weather Event Explorer #22