INCIDENT: production has delivered no alerts since 2026-08-11 — crash-looping on a stale image against a forward-migrated DB #186

New issue

Open

opened 2026-08-20 19:17:53 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-08-20 19:17:53 +00:00

Contributor

Severity: high. Production severe-weather alerting has been silently down for ~9.5 days. Found incidentally on 2026-08-20 while querying production for an unrelated radar_animated distribution (#184).

Impact

Last alert delivered: 2026-08-11 03:27:05 UTC. Nothing since. sent_alerts holds 1,719 rows and has not grown in 9.5 days.
The app container has restarted 13,294 times and is still restarting.
Nothing detected or reported this. The stack's other containers (postgres, signal-cli) are Up 9 days (healthy), so a container-level health view looks partly fine.

Root cause

The database is at Alembic revision 0033 (shipped in v1.8.1). The container is running weatherbot-app — a locally-built image reporting WEATHERBOT_VERSION=v1.7.0-29-g12d47dd, which predates 0033 and does not contain it. Startup therefore dies every time:

ERROR [alembic.util.messaging] Can't locate revision identified by '0033'
FAILED: Can't locate revision identified by '0033'
subprocess.CalledProcessError: Command '[... '-m', 'alembic', 'upgrade', 'head']' returned non-zero exit status 255
ERROR:    Application startup failed. Exiting.

app/main.py's lifespan runs alembic upgrade head with check=True, so a migration failure is fatal to startup by design — correct behaviour, but it means a code/DB mismatch is an unrecoverable crash loop rather than a degraded start.

How the mismatch happened

The running container's compose labels point at the old pre-Ansible checkout:

com.docker.compose.project             = weatherbot
com.docker.compose.project.config_files = /home/ryan/WeatherBot/docker-compose.yml
com.docker.compose.project.working_dir  = /home/ryan/WeatherBot

while the Ansible-managed deployment lives at /opt/weatherbot-svc. Both use the compose project name weatherbot, so they own the same containers. Something ran docker compose up from /home/ryan/WeatherBot on ~2026-08-11 05:54 (container Created timestamp), which rebuilt the app from that stale checkout and replaced the Ansible-deployed v1.8.1 container — while leaving the database forward-migrated at 0033.

This is exactly the failure mode .forgejo/workflows/cd.yml documents as the reason the merge-time dev deploy was retired: "it rebuilt the stack from a git checkout under the same compose project name the Ansible role owns." The same landmine existed on production and was never disarmed.

weatherbot.service (the Ansible-managed systemd unit) is present but in failed state.

Proposed remediation

Restore service — run the weatherbot play against docker-host. The group_vars pin is 1.8.1, whose schema is exactly 0033, so it matches the database with no migration needed. This is the committed intended state.
Disarm the landmine — /home/ryan/WeatherBot must stop being able to clobber the Ansible stack. Either remove it, or rename its compose project so it can never again own weatherbot's containers. Restoring service without this leaves the same trap armed.
Detection gap — 9.5 days of total outage with no alarm is the more serious finding than the outage itself. The poller heartbeat (I-07) detects upstream staleness, but nothing noticed the app was not running at all. Needs its own issue.

Not the cause

The weatherbot-backup.timer is active and its service inactive dead (normal between runs). Postgres and signal-cli are healthy. This is purely the app container.

Incidental finding (the query that surfaced this)

notification_channels on production: 4 rows, all radar_animated = true. So #184's migration 0037 backfill is a no-op on production as well as dev — no channel has opted out of radar, so none can be wrongly switched on. The migration remains correct; there is simply no row that exercises it.

**Severity: high. Production severe-weather alerting has been silently down for ~9.5 days.** Found incidentally on 2026-08-20 while querying production for an unrelated `radar_animated` distribution (#184). ## Impact - **Last alert delivered: `2026-08-11 03:27:05 UTC`.** Nothing since. `sent_alerts` holds 1,719 rows and has not grown in 9.5 days. - The app container has restarted **13,294 times** and is still restarting. - Nothing detected or reported this. The stack's other containers (`postgres`, `signal-cli`) are `Up 9 days (healthy)`, so a container-level health view looks partly fine. ## Root cause The database is at Alembic revision **`0033`** (shipped in v1.8.1). The container is running **`weatherbot-app`** — a locally-built image reporting `WEATHERBOT_VERSION=v1.7.0-29-g12d47dd`, which predates `0033` and does not contain it. Startup therefore dies every time: ``` ERROR [alembic.util.messaging] Can't locate revision identified by '0033' FAILED: Can't locate revision identified by '0033' subprocess.CalledProcessError: Command '[... '-m', 'alembic', 'upgrade', 'head']' returned non-zero exit status 255 ERROR: Application startup failed. Exiting. ``` `app/main.py`'s lifespan runs `alembic upgrade head` with `check=True`, so a migration failure is fatal to startup by design — correct behaviour, but it means a code/DB mismatch is an unrecoverable crash loop rather than a degraded start. ### How the mismatch happened The running container's compose labels point at the **old pre-Ansible checkout**: ``` com.docker.compose.project = weatherbot com.docker.compose.project.config_files = /home/ryan/WeatherBot/docker-compose.yml com.docker.compose.project.working_dir = /home/ryan/WeatherBot ``` while the Ansible-managed deployment lives at `/opt/weatherbot-svc`. **Both use the compose project name `weatherbot`**, so they own the same containers. Something ran `docker compose up` from `/home/ryan/WeatherBot` on ~2026-08-11 05:54 (container `Created` timestamp), which rebuilt the app from that stale checkout and replaced the Ansible-deployed v1.8.1 container — while leaving the database forward-migrated at `0033`. This is exactly the failure mode `.forgejo/workflows/cd.yml` documents as the reason the merge-time dev deploy was retired: *"it rebuilt the stack from a git checkout under the same compose project name the Ansible role owns."* The same landmine existed on production and was never disarmed. `weatherbot.service` (the Ansible-managed systemd unit) is present but in **`failed`** state. ## Proposed remediation 1. **Restore service** — run the weatherbot play against `docker-host`. The `group_vars` pin is `1.8.1`, whose schema is exactly `0033`, so it matches the database with no migration needed. This is the committed intended state. 2. **Disarm the landmine** — `/home/ryan/WeatherBot` must stop being able to clobber the Ansible stack. Either remove it, or rename its compose project so it can never again own `weatherbot`'s containers. Restoring service without this leaves the same trap armed. 3. **Detection gap** — 9.5 days of total outage with no alarm is the more serious finding than the outage itself. The poller heartbeat (I-07) detects *upstream* staleness, but nothing noticed the app was not running at all. Needs its own issue. ## Not the cause The `weatherbot-backup.timer` is `active` and its service `inactive dead` (normal between runs). Postgres and signal-cli are healthy. This is purely the app container. ## Incidental finding (the query that surfaced this) `notification_channels` on production: **4 rows, all `radar_animated = true`**. So #184's migration `0037` backfill is a no-op on production as well as dev — no channel has opted out of radar, so none can be wrongly switched on. The migration remains correct; there is simply no row that exercises it.

claude-bot commented

2026-08-20 22:03:18 +00:00

Author

Contributor

Resolved — service restored 2026-08-20 ~21:47 UTC

Production is running git.rhoving.com/rbrooks/weatherbot:1.8.1 (the registry image the pin specifies, not a local build), healthy, with the full Ansible-managed stack recreated (postgres:16.14-alpine, signal-cli).

Confirmed working end-to-end, not merely container-healthy: all APScheduler jobs execute successfully (poll_alerts_job, poll_spc_job, process_delivery_outbox_job, …), and sent_alerts recorded a delivery at 2026-08-20 21:51:25 UTC — the first since 2026-08-11 03:27. Total outage: ~9.7 days. Database remains at 0033, matching 1.8.1, so no migration was needed.

Correcting the root cause above

My original attribution — "something ran docker compose up from /home/ryan/WeatherBot" — was stated with more confidence than the evidence supported. The journal now shows a more specific first domino:

Aug 11 05:39:38  Container weatherbot-db-1 Starting
Aug 11 05:40:44  Container weatherbot-db-1 Started          <- 66s to start
Aug 11 05:40:51  Container weatherbot-db-1 Waiting          <- waiting on health
Aug 11 05:40:59  weatherbot.service: start operation timed out. Terminating.
Aug 11 05:40:59  Main process exited, code=exited, status=130   (SIGINT)
Aug 11 05:40:59  Failed with result 'timeout'

weatherbot.service timed out during docker compose up because Postgres took 66s to become healthy, and systemd SIGINT'd compose with the stack half-started — the app container never came up under the unit.

This is precisely the failure the systemd-unit change in iac-repo#277 was written to fix, whose own comment reads: "is slow to become healthy, compose up stops waiting, and the unit dies with the stack half-started." Production had not received that change.

What happened between 05:40:59 (unit failed) and 05:54:48 (the crash-looping weatherbot-app container's creation timestamp) is not established by evidence I have. The 14-minute gap and the container's build-from-source origin at /home/ryan/WeatherBot are consistent with a manual recovery attempt from the old pre-Ansible checkout, but I did not find a log proving that, and I should not have asserted it as fact. What is certain is that the resulting container was built from stale v1.7.0-era source and could not satisfy the database's 0033, producing 13,294 restarts.

So the chain was: iac-repo#277 missing on prod → compose-up timeout on slow Postgres → half-started stack → recovery from a stale checkout → permanent version/schema mismatch.

Still outstanding

weatherbot.service remains in failed state (stale, from Aug 11) while the containers run healthy outside it. The unit file is now correct — WorkingDirectory=/opt/weatherbot-svc, TimeoutStartSec=300, and #277's Restart=on-failure / RestartSec=30 are all present — so this is leftover state, not a live fault. It wants systemctl reset-failed weatherbot.service followed by systemctl start weatherbot.service so the unit adopts the running stack. Until then, anything monitoring unit state reports a failure that no longer exists.
The landmine is still armed. /home/ryan/WeatherBot/docker-compose.yml still exists on docker-host, and weatherbot-app:latest is still in the local image store. Both should go, or the checkout's compose project renamed, so it can never again own the weatherbot containers.
The detection gap is the most important finding. ~9.7 days of complete outage on a life-safety service with nothing raising an alarm. The poller heartbeat detects upstream staleness; nothing detects "the app is not running." This deserves its own issue and should arguably gate v2.0.0.

## Resolved — service restored 2026-08-20 ~21:47 UTC Production is running `git.rhoving.com/rbrooks/weatherbot:1.8.1` (the registry image the pin specifies, not a local build), healthy, with the full Ansible-managed stack recreated (`postgres:16.14-alpine`, signal-cli). **Confirmed working end-to-end, not merely container-healthy:** all APScheduler jobs execute successfully (`poll_alerts_job`, `poll_spc_job`, `process_delivery_outbox_job`, …), and **`sent_alerts` recorded a delivery at `2026-08-20 21:51:25 UTC`** — the first since 2026-08-11 03:27. Total outage: **~9.7 days**. Database remains at `0033`, matching 1.8.1, so no migration was needed. ## Correcting the root cause above My original attribution — "something ran `docker compose up` from `/home/ryan/WeatherBot`" — was stated with more confidence than the evidence supported. The journal now shows a more specific first domino: ``` Aug 11 05:39:38 Container weatherbot-db-1 Starting Aug 11 05:40:44 Container weatherbot-db-1 Started <- 66s to start Aug 11 05:40:51 Container weatherbot-db-1 Waiting <- waiting on health Aug 11 05:40:59 weatherbot.service: start operation timed out. Terminating. Aug 11 05:40:59 Main process exited, code=exited, status=130 (SIGINT) Aug 11 05:40:59 Failed with result 'timeout' ``` **`weatherbot.service` timed out during `docker compose up` because Postgres took 66s to become healthy**, and systemd SIGINT'd compose with the stack half-started — the app container never came up under the unit. This is precisely the failure the systemd-unit change in **iac-repo#277** was written to fix, whose own comment reads: *"is slow to become healthy, `compose up` stops waiting, and the unit dies with the stack half-started."* Production had not received that change. What happened between `05:40:59` (unit failed) and `05:54:48` (the crash-looping `weatherbot-app` container's creation timestamp) is **not established by evidence I have**. The 14-minute gap and the container's build-from-source origin at `/home/ryan/WeatherBot` are consistent with a manual recovery attempt from the old pre-Ansible checkout, but I did not find a log proving that, and I should not have asserted it as fact. What is certain is that the resulting container was built from stale v1.7.0-era source and could not satisfy the database's `0033`, producing 13,294 restarts. So the chain was: **iac-repo#277 missing on prod → compose-up timeout on slow Postgres → half-started stack → recovery from a stale checkout → permanent version/schema mismatch.** ## Still outstanding 1. **`weatherbot.service` remains in `failed` state** (stale, from Aug 11) while the containers run healthy outside it. The unit file is now correct — `WorkingDirectory=/opt/weatherbot-svc`, `TimeoutStartSec=300`, and #277's `Restart=on-failure` / `RestartSec=30` are all present — so this is leftover state, not a live fault. It wants `systemctl reset-failed weatherbot.service` followed by `systemctl start weatherbot.service` so the unit adopts the running stack. Until then, anything monitoring unit state reports a failure that no longer exists. 2. **The landmine is still armed.** `/home/ryan/WeatherBot/docker-compose.yml` still exists on `docker-host`, and `weatherbot-app:latest` is still in the local image store. Both should go, or the checkout's compose project renamed, so it can never again own the `weatherbot` containers. 3. **The detection gap is the most important finding.** ~9.7 days of complete outage on a life-safety service with nothing raising an alarm. The poller heartbeat detects upstream staleness; nothing detects "the app is not running." This deserves its own issue and should arguably gate v2.0.0.

claude-bot referenced this issue

2026-08-21 03:43:35 +00:00

Notifier radar: make "animated" actually animate, and split the flag that doesn't mean what it says #184

claude-bot referenced this issue

2026-08-21 03:50:32 +00:00

Historical Weather Event Explorer #22