rbrooks/TeaLeaves

Watch

Fork 0

Add API healthcheck and deepen /health to probe DB + Redis #97

New issue

Closed

opened 2026-07-15 19:51:31 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-15 19:51:31 +00:00

Contributor

Severity: Medium · Confidence: High · Effort: S · Category: ops

Evidence

docker-compose.yml - api service has restart: unless-stopped, no healthcheck; app depends_on: api with no condition.
api/src/app.ts:81-83 - /health returns {status:'ok'} unconditionally; no DB/Redis probe.
api/Dockerfile - no HEALTHCHECK.

Problem
A wedged-but-alive API (pg pool exhausted, event loop blocked, e.g. by F-09) is never detected or restarted; restart: unless-stopped only reacts to process exit. /health can't distinguish "up" from "up but DB down."

Impact
Silent brownouts with no auto-recovery.

Fix
Add a compose healthcheck hitting /health; deepen /health to run SELECT 1 + Redis PING with a short timeout; gate app on service_healthy.

Acceptance criteria

/health returns non-200 when DB or Redis is unreachable.
Compose restarts the api container when the healthcheck fails.

Related: F-09 (backup hang can wedge the event loop).

Filed from the 2026-07-15 codebase audit. Full report: docs/.internal/report-2026-07-15.md (gitignored).

**Severity:** Medium · **Confidence:** High · **Effort:** S · Category: ops **Evidence** - `docker-compose.yml` - api service has `restart: unless-stopped`, no `healthcheck`; app `depends_on: api` with no condition. - `api/src/app.ts:81-83` - `/health` returns `{status:'ok'}` unconditionally; no DB/Redis probe. - `api/Dockerfile` - no `HEALTHCHECK`. **Problem** A wedged-but-alive API (pg pool exhausted, event loop blocked, e.g. by F-09) is never detected or restarted; `restart: unless-stopped` only reacts to process exit. `/health` can't distinguish "up" from "up but DB down." **Impact** Silent brownouts with no auto-recovery. **Fix** Add a compose healthcheck hitting `/health`; deepen `/health` to run `SELECT 1` + Redis `PING` with a short timeout; gate `app` on `service_healthy`. **Acceptance criteria** - [ ] `/health` returns non-200 when DB or Redis is unreachable. - [ ] Compose restarts the api container when the healthcheck fails. Related: F-09 (backup hang can wedge the event loop). --- _Filed from the 2026-07-15 codebase audit. Full report: `docs/.internal/report-2026-07-15.md` (gitignored)._

claude-bot added this to the v7.2.0 - Reliability, data integrity & docs milestone

2026-07-15 19:51:31 +00:00

claude-bot added the

enhancement

label

2026-07-15 19:51:31 +00:00

rbrooks referenced this issue from a commit

2026-07-17 03:18:43 +00:00

feat(v7.2): deepen /health + build production images in CI (#97, #103)

claude-bot referenced this issue

2026-07-17 03:19:15 +00:00

Nothing restarts the API when it goes unhealthy — Docker Compose does not act on healthcheck status #137

claude-bot referenced this issue

2026-07-17 03:20:58 +00:00

Auto-deploy: ship main to dev and tagged releases to prod (no more hand-deploys) #134

claude-bot commented

2026-07-17 03:39:10 +00:00

Author

Contributor

Done in 109c53a (v7.2.0) — with one criterion that cannot be met as written; details below.

`/health` now probes its dependencies

SELECT 1 on the pg pool and a Redis PING, each wrapped in a 2 s timeout so a hung dependency can't hang the endpoint itself. Returns 503 with per-dependency detail. Verified live on the dev server by stopping Redis:

healthy:     HTTP 200  {"status":"ok","db":"ok","redis":"ok"}
redis down:  HTTP 503  {"status":"error","db":"ok","redis":"error: probe timed out after 2000ms"}
restored:    HTTP 200  {"status":"ok","db":"ok","redis":"ok"}

Two things worth recording:

/health was registered after defaultRateLimit. A container healthcheck polls continuously; a 429 would be read as "unhealthy" and could restart-loop a perfectly good container. It's now registered before the rate limiter — the limiter must never be able to fail the healthcheck.
The API image has no curl and no wget (the Dockerfile purges curl after fetching the PGDG key), so the conventional HEALTHCHECK CMD curl -f localhost:3000/health would never have worked here. The HEALTHCHECK uses node's global fetch instead — no new packages, no image bloat.

app is now gated on condition: service_healthy, so nginx stops proxying to an API that's still running migrations. Visible in the deploy output: Container tealeaves-api-1 Healthy before app starts.

Acceptance criteria

/health returns non-200 when DB or Redis is unreachable.
~~Compose restarts the api container when the healthcheck fails.~~ — not achievable; the premise is wrong.

Docker Compose does not restart containers on healthcheck failure. restart: reacts to the process exiting; acting on health status is Swarm/Kubernetes behaviour, and there is no restart_on_unhealthy in Compose. Verified rather than assumed — stopped Redis and waited out the 3 retries:

RestartCount before: 0
Health=unhealthy  RestartCount=0  Running=true
tealeaves-api-1  Up 3 minutes (unhealthy)

Detected, marked unhealthy, never restarted.

Closing because the title and the substantive work are done — the probe, the HEALTHCHECK, and the depends_on gating. Auto-recovery is split out to #137, which needs an infrastructure decision rather than more code: the usual answer (an autoheal sidecar) wants /var/run/docker.sock mounted, i.e. root-equivalent access on a host running ~55 other containers — not a trade to make silently for a rare brownout. Four options laid out there.

CI green on re-run of 109c53a (249/249). Deployed to dev, container reporting healthy.

Done in `109c53a` (v7.2.0) — with **one criterion that cannot be met as written**; details below. ### `/health` now probes its dependencies `SELECT 1` on the pg pool and a Redis `PING`, each wrapped in a 2 s timeout so a *hung* dependency can't hang the endpoint itself. Returns 503 with per-dependency detail. Verified live on the dev server by stopping Redis: ``` healthy: HTTP 200 {"status":"ok","db":"ok","redis":"ok"} redis down: HTTP 503 {"status":"error","db":"ok","redis":"error: probe timed out after 2000ms"} restored: HTTP 200 {"status":"ok","db":"ok","redis":"ok"} ``` Two things worth recording: - **`/health` was registered *after* `defaultRateLimit`.** A container healthcheck polls continuously; a 429 would be read as "unhealthy" and could restart-loop a perfectly good container. It's now registered *before* the rate limiter — the limiter must never be able to fail the healthcheck. - **The API image has no `curl` and no `wget`** (the Dockerfile purges curl after fetching the PGDG key), so the conventional `HEALTHCHECK CMD curl -f localhost:3000/health` would never have worked here. The `HEALTHCHECK` uses node's global `fetch` instead — no new packages, no image bloat. `app` is now gated on `condition: service_healthy`, so nginx stops proxying to an API that's still running migrations. Visible in the deploy output: `Container tealeaves-api-1 Healthy` before app starts. ### Acceptance criteria - [x] `/health` returns non-200 when DB or Redis is unreachable. - [ ] ~~Compose restarts the api container when the healthcheck fails.~~ — **not achievable; the premise is wrong.** **Docker Compose does not restart containers on healthcheck failure.** `restart:` reacts to the process *exiting*; acting on health status is Swarm/Kubernetes behaviour, and there is no `restart_on_unhealthy` in Compose. Verified rather than assumed — stopped Redis and waited out the 3 retries: ``` RestartCount before: 0 Health=unhealthy RestartCount=0 Running=true tealeaves-api-1 Up 3 minutes (unhealthy) ``` Detected, marked unhealthy, never restarted. Closing because the title and the substantive work are done — the probe, the `HEALTHCHECK`, and the `depends_on` gating. **Auto-recovery is split out to #137**, which needs an infrastructure decision rather than more code: the usual answer (an autoheal sidecar) wants `/var/run/docker.sock` mounted, i.e. root-equivalent access on a host running ~55 other containers — not a trade to make silently for a rare brownout. Four options laid out there. CI green on re-run of `109c53a` (249/249). Deployed to dev, container reporting `healthy`.

claude-bot closed this issue

2026-07-17 03:39:26 +00:00

claude-bot referenced this issue

2026-07-17 16:15:31 +00:00

Auto-deploy: ship main to dev and tagged releases to prod (no more hand-deploys) #134

claude-bot referenced this issue

2026-07-17 16:17:16 +00:00

Nothing restarts the API when it goes unhealthy — Docker Compose does not act on healthcheck status #137

rbrooks referenced this issue from a commit

2026-07-17 20:27:24 +00:00

feat(deploy): auto-deploy scaffolding (#134) + image-dedup upload warning (#136)