rbrooks/TeaLeaves

Watch

Fork 0

Nothing restarts the API when it goes unhealthy — Docker Compose does not act on healthcheck status #137

New issue

Closed

opened 2026-07-17 03:19:15 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-07-17 03:19:15 +00:00

Contributor

Severity: Medium · Confidence: High (verified empirically) · Effort: S · Category: ops

Problem

#97 added a real /health probe (DB + Redis) and a container HEALTHCHECK, so a wedged-but-running API is now detected. Nothing acts on it.

restart: unless-stopped only reacts to the process exiting. Docker Engine marks a container unhealthy and takes no further action — restarting on health status is Swarm/Kubernetes behaviour, and there is no restart_on_unhealthy in Compose.

So #97's second acceptance criterion ("Compose restarts the api container when the healthcheck fails") rests on a mistaken premise about Docker and cannot be met by configuration alone.

Evidence (dev server, 2026-07-17)

Stopped Redis and waited for the healthcheck to fail its 3 retries:

RestartCount before: 0
redis stopped; waiting ~110s ...
Health=unhealthy  RestartCount=0  Running=true
tealeaves-api-1  Up 3 minutes (unhealthy)

The container sat unhealthy indefinitely, still serving 503s, and was never restarted.

Impact

The exact scenario #97 set out to fix — "a wedged-but-alive API is never detected or restarted" — is now half-solved. It's detected (docker compose ps shows (unhealthy), and depends_on: service_healthy respects it), but a silent brownout still requires someone to notice and docker compose restart api by hand.

Options

autoheal sidecar (willfarrell/autoheal) — watches for unhealthy containers and restarts them; the standard answer for plain Compose. Security cost: it needs /var/run/docker.sock mounted, which is root-equivalent on the host. That's a real decision for a box also running ~55 other containers, and shouldn't be taken by default.
Self-terminate on sustained unhealth — have the API detect its own wedged state and process.exit(1), letting restart: unless-stopped do the work. No new privileges, no new container; but it means the app judges its own health, and a flapping dependency could turn into a restart loop. Would want a generous threshold and to reuse the #85 graceful-shutdown path.
External monitoring — the homelab already has observability (Rhoving/iac-repo); alert on unhealthy and let a human decide. No auto-recovery, but no new attack surface either.
Accept it. The healthcheck's value is depends_on gating and visibility; a wedged API is rare enough to fix by hand.

My weak preference is (2) or (3) over (1) — mounting the docker socket to fix a rare brownout is a poor trade on a host with that much else on it. But this is a judgement call about your infrastructure, not a code question.

Acceptance criteria

A decision is recorded on how (or whether) an unhealthy API recovers automatically.
(If 1 or 2) an API wedged by an unreachable dependency returns to service without human intervention, and cannot restart-loop while the dependency is simply down.

Notes

Split out of #97, which delivered the probe, the HEALTHCHECK and the depends_on: service_healthy gating. Filed rather than folded in because the fix needs new infrastructure and a security trade-off, not more code. Related: #134 (auto-deploy) gates on /health, so it benefits from the probe regardless of what's decided here.

**Severity:** Medium · **Confidence:** High (verified empirically) · **Effort:** S · Category: ops ## Problem #97 added a real `/health` probe (DB + Redis) and a container `HEALTHCHECK`, so a wedged-but-running API is now **detected**. Nothing **acts** on it. `restart: unless-stopped` only reacts to the process *exiting*. Docker Engine marks a container `unhealthy` and takes no further action — restarting on health status is Swarm/Kubernetes behaviour, and there is no `restart_on_unhealthy` in Compose. So #97's second acceptance criterion ("Compose restarts the api container when the healthcheck fails") rests on a mistaken premise about Docker and cannot be met by configuration alone. ## Evidence (dev server, 2026-07-17) Stopped Redis and waited for the healthcheck to fail its 3 retries: ``` RestartCount before: 0 redis stopped; waiting ~110s ... Health=unhealthy RestartCount=0 Running=true tealeaves-api-1 Up 3 minutes (unhealthy) ``` The container sat `unhealthy` indefinitely, still serving 503s, and was never restarted. ## Impact The exact scenario #97 set out to fix — "a wedged-but-alive API is never detected **or restarted**" — is now half-solved. It's detected (`docker compose ps` shows `(unhealthy)`, and `depends_on: service_healthy` respects it), but a silent brownout still requires someone to notice and `docker compose restart api` by hand. ## Options 1. **autoheal sidecar** (`willfarrell/autoheal`) — watches for `unhealthy` containers and restarts them; the standard answer for plain Compose. **Security cost:** it needs `/var/run/docker.sock` mounted, which is root-equivalent on the host. That's a real decision for a box also running ~55 other containers, and shouldn't be taken by default. 2. **Self-terminate on sustained unhealth** — have the API detect its own wedged state and `process.exit(1)`, letting `restart: unless-stopped` do the work. No new privileges, no new container; but it means the app judges its own health, and a flapping dependency could turn into a restart loop. Would want a generous threshold and to reuse the #85 graceful-shutdown path. 3. **External monitoring** — the homelab already has observability (`Rhoving/iac-repo`); alert on `unhealthy` and let a human decide. No auto-recovery, but no new attack surface either. 4. **Accept it.** The healthcheck's value is `depends_on` gating and visibility; a wedged API is rare enough to fix by hand. My weak preference is (2) or (3) over (1) — mounting the docker socket to fix a rare brownout is a poor trade on a host with that much else on it. But this is a judgement call about your infrastructure, not a code question. ## Acceptance criteria - [ ] A decision is recorded on how (or whether) an unhealthy API recovers automatically. - [ ] (If 1 or 2) an API wedged by an unreachable dependency returns to service without human intervention, and cannot restart-loop while the dependency is simply down. ## Notes Split out of #97, which delivered the probe, the `HEALTHCHECK` and the `depends_on: service_healthy` gating. Filed rather than folded in because the fix needs new infrastructure and a security trade-off, not more code. Related: #134 (auto-deploy) gates on `/health`, so it benefits from the probe regardless of what's decided here.

claude-bot referenced this issue

2026-07-17 03:39:10 +00:00

Add API healthcheck and deepen /health to probe DB + Redis #97

claude-bot commented

2026-07-17 16:00:39 +00:00

Author

Contributor

Decision (from @rbrooks): autoheal is out. Recording per this issue's acceptance criterion.

So option 1 (the willfarrell/autoheal sidecar) is rejected — it would need /var/run/docker.sock mounted (root-equivalent) on a host running ~55 other containers, too much standing attack surface to fix a rare brownout.

That leaves the two socket-free options. My recommendation between them:

Option 3 — external monitoring (recommended default). Zero new code, zero new attack surface. The homelab already has observability (Rhoving/iac-repo); alert on a container in unhealthy state (or scrape /health → non-200) and page a human. Given how rare a wedged-but-alive API is — and that depends_on: service_healthy already prevents the app from proxying to a not-yet-ready API — visibility + an alert is a proportionate answer.
Option 2 — API self-terminates on sustained unhealth — the move only if you want hands-off auto-recovery without the socket. Small, self-contained: a periodic self-probe (reuse the /health checks) that, after N consecutive failures over a generous window, calls the existing #85 graceful-shutdown path with a non-zero exit so restart: unless-stopped brings it back clean. The risk to design against is a restart-loop while a dependency is simply down (Redis/DB unreachable isn't the API's fault) — so the threshold must be generous and ideally distinguish "I am wedged" from "my dependency is absent".

Proposed: go with option 3 now (it's a config/alerting change in iac-repo, not TeaLeaves code), and hold option 2 as a fast-follow if a real brownout ever actually bites. Leaving this issue open for the monitoring wiring; happy to implement option 2 in TeaLeaves instead if you'd rather have auto-recovery — your call.

**Decision (from @rbrooks): autoheal is out.** Recording per this issue's acceptance criterion. So option 1 (the `willfarrell/autoheal` sidecar) is rejected — it would need `/var/run/docker.sock` mounted (root-equivalent) on a host running ~55 other containers, too much standing attack surface to fix a rare brownout. That leaves the two socket-free options. My recommendation between them: - **Option 3 — external monitoring (recommended default).** Zero new code, zero new attack surface. The homelab already has observability (`Rhoving/iac-repo`); alert on a container in `unhealthy` state (or scrape `/health` → non-200) and page a human. Given how rare a wedged-but-alive API is — and that `depends_on: service_healthy` already prevents the app from proxying to a not-yet-ready API — visibility + an alert is a proportionate answer. - **Option 2 — API self-terminates on sustained unhealth** — the move *only if* you want hands-off auto-recovery without the socket. Small, self-contained: a periodic self-probe (reuse the `/health` checks) that, after N consecutive failures over a generous window, calls the existing #85 graceful-shutdown path with a non-zero exit so `restart: unless-stopped` brings it back clean. The risk to design against is a restart-loop while a dependency is simply *down* (Redis/DB unreachable isn't the API's fault) — so the threshold must be generous and ideally distinguish "I am wedged" from "my dependency is absent". **Proposed:** go with **option 3** now (it's a config/alerting change in iac-repo, not TeaLeaves code), and hold option 2 as a fast-follow if a real brownout ever actually bites. Leaving this issue open for the monitoring wiring; happy to implement option 2 in TeaLeaves instead if you'd rather have auto-recovery — your call.

claude-bot referenced this issue

2026-07-17 16:15:31 +00:00

Auto-deploy: ship main to dev and tagged releases to prod (no more hand-deploys) #134

claude-bot commented

2026-07-17 16:17:16 +00:00

Author

Contributor

Decided: option 3 (external monitoring), tracked in Rhoving/iac-repo#163.

@rbrooks picked external monitoring over the self-terminate option, and will implement it in the IaC repo. Opened Rhoving/iac-repo#163 with the full details: a blackbox HTTP probe of the /health endpoints (prod 10.1.1.7:8055, dev 10.1.1.14:18090) scraped by the existing Prometheus stack, with an Alertmanager rule firing when /health returns non-2xx — no docker.sock, no agent on the app hosts.

The TeaLeaves side of this is already done and shipped in v7.2.0: the real /health DB+Redis probe (#97) that returns 503 when degraded is exactly the signal the monitor consumes. Nothing further to build here.

Closing — decision recorded (this issue's acceptance criterion), implementation handed to iac-repo#163.

**Decided: option 3 (external monitoring), tracked in Rhoving/iac-repo#163.** @rbrooks picked external monitoring over the self-terminate option, and will implement it in the IaC repo. Opened **Rhoving/iac-repo#163** with the full details: a blackbox HTTP probe of the `/health` endpoints (prod `10.1.1.7:8055`, dev `10.1.1.14:18090`) scraped by the existing Prometheus stack, with an Alertmanager rule firing when `/health` returns non-2xx — no docker.sock, no agent on the app hosts. The TeaLeaves side of this is already done and shipped in v7.2.0: the real `/health` DB+Redis probe (#97) that returns 503 when degraded is exactly the signal the monitor consumes. Nothing further to build here. Closing — decision recorded (this issue's acceptance criterion), implementation handed to iac-repo#163.

claude-bot closed this issue

2026-07-17 16:17:17 +00:00

claude-bot referenced this issue

2026-07-17 21:52:31 +00:00

Auto-deploy: ship main to dev and tagged releases to prod (no more hand-deploys) #134