rbrooks/TeaLeaves

Watch

Fork 0

worker container is permanently `unhealthy` — it inherits the API image's HTTP healthcheck but serves no HTTP #156

New issue

Open

opened 2026-08-17 19:35:03 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-08-17 19:35:03 +00:00

Contributor

Found immediately after the #155 cutover. Both hosts show it:

tealeaves-worker-1   ...tealeaves-api:9.4.0   Up 52 minutes (unhealthy)    # docker-host
tealeaves-worker-1   ...tealeaves-api:beta    Up 58 minutes (unhealthy)    # ai-dev-server

Cause

The worker service runs the API image with a different command (node dist/worker.js), so it inherits that image's HEALTHCHECK — api/Dockerfile:41:

CMD node -e "fetch('http://127.0.0.1:3000/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"

The worker process does not listen on 3000. It has no HTTP surface at all. So the probe cannot ever succeed:

$ docker inspect -f '{{range .State.Health.Log}}exit={{.ExitCode}}{{end}}' tealeaves-worker-1
exit=1 exit=1 exit=1 exit=1 exit=1

The worker itself is fine — this is a false negative, not a broken worker:

Tea Leaves worker process started (11 workers)
[watch] worker started; polling every 10 min
[trash] Running purge — retention: 30 days
[trash] Purge complete — entries: 0, motifs: 0, projects: 0

Why it's latent until now

#117 added the worker service, but nothing ever synced the compose file to a host (#155) — so this is the first time the worker container has ever run anywhere. The defect shipped with #117 and has been invisible for a month.

Why it matters

Nothing depends on the worker via condition: service_healthy, so there is no cascade and no outage. The cost is monitoring integrity:

A container that is red by construction and always will be trains everyone to ignore red containers — including the next one that means something.
Rhoving/iac-repo has an open issue for a blackbox /health probe + Prometheus alerting on this stack, and cAdvisor-based container-health alerting would fire on this permanently.
It also makes "is the worker actually running?" unanswerable by the cheapest available signal, which matters because a silently dead worker is exactly the failure this would otherwise catch — imports, media processing, embeddings, backups and federation delivery all stop with no user-visible error.

Options

Disable the inherited healthcheck for the worker service (healthcheck: {disable: true} in the Ansible compose template). Honest and one line — the container genuinely has no HTTP surface. But it trades a false red for no signal at all, and per the point above, "no signal" is what makes a dead worker invisible.
Give the worker a real liveness check. Either a tiny HTTP listener in worker.ts on its own port serving /health, or a CMD that checks BullMQ liveness (e.g. the worker heartbeats a Redis key and the probe asserts freshness). More work, and it's the one that actually answers the question.
Split the Dockerfile healthcheck out of the shared image so the API compose service declares it rather than the image. Cleanest separation, but touches the API's own health story.

Leaning (2) with a Redis heartbeat — it needs no new port and directly tests the thing that matters (the worker is consuming). (1) is a reasonable stopgap in the same change if (2) isn't wanted now, but shipping (1) alone should be a deliberate decision, not the default.

The fix lands in this repo (Dockerfile and/or worker.ts); the compose-side half is in Rhoving/iac-repo → tealeaves-ansible/roles/tealeaves/templates/docker-compose.yml.j2.

Refs #117, #155, Rhoving/iac-repo#319

Found immediately after the #155 cutover. **Both hosts** show it: ``` tealeaves-worker-1 ...tealeaves-api:9.4.0 Up 52 minutes (unhealthy) # docker-host tealeaves-worker-1 ...tealeaves-api:beta Up 58 minutes (unhealthy) # ai-dev-server ``` ## Cause The `worker` service runs the **API image** with a different command (`node dist/worker.js`), so it inherits that image's `HEALTHCHECK` — [api/Dockerfile:41](api/Dockerfile#L41): ``` CMD node -e "fetch('http://127.0.0.1:3000/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))" ``` The worker process does not listen on 3000. It has no HTTP surface at all. So the probe cannot ever succeed: ``` $ docker inspect -f '{{range .State.Health.Log}}exit={{.ExitCode}}{{end}}' tealeaves-worker-1 exit=1 exit=1 exit=1 exit=1 exit=1 ``` **The worker itself is fine** — this is a false negative, not a broken worker: ``` Tea Leaves worker process started (11 workers) [watch] worker started; polling every 10 min [trash] Running purge — retention: 30 days [trash] Purge complete — entries: 0, motifs: 0, projects: 0 ``` ## Why it's latent until now #117 added the `worker` service, but nothing ever synced the compose file to a host (#155) — so **this is the first time the worker container has ever run anywhere**. The defect shipped with #117 and has been invisible for a month. ## Why it matters Nothing depends on the worker via `condition: service_healthy`, so there is no cascade and no outage. The cost is monitoring integrity: - A container that is red *by construction* and always will be trains everyone to ignore red containers — including the next one that means something. - `Rhoving/iac-repo` has an open issue for a blackbox `/health` probe + Prometheus alerting on this stack, and cAdvisor-based container-health alerting would fire on this permanently. - It also makes "is the worker actually running?" unanswerable by the cheapest available signal, which matters because a silently dead worker is exactly the failure this would otherwise catch — imports, media processing, embeddings, backups and federation delivery all stop with no user-visible error. ## Options 1. **Disable the inherited healthcheck for the `worker` service** (`healthcheck: {disable: true}` in the Ansible compose template). Honest and one line — the container genuinely has no HTTP surface. But it trades a false red for no signal at all, and per the point above, "no signal" is what makes a dead worker invisible. 2. **Give the worker a real liveness check.** Either a tiny HTTP listener in `worker.ts` on its own port serving `/health`, or a `CMD` that checks BullMQ liveness (e.g. the worker heartbeats a Redis key and the probe asserts freshness). More work, and it's the one that actually answers the question. 3. **Split the Dockerfile healthcheck out of the shared image** so the API compose service declares it rather than the image. Cleanest separation, but touches the API's own health story. Leaning (2) with a Redis heartbeat — it needs no new port and directly tests the thing that matters (the worker is consuming). (1) is a reasonable stopgap in the same change if (2) isn't wanted now, but shipping (1) alone should be a deliberate decision, not the default. The fix lands in this repo (Dockerfile and/or `worker.ts`); the compose-side half is in `Rhoving/iac-repo` → `tealeaves-ansible/roles/tealeaves/templates/docker-compose.yml.j2`. Refs #117, #155, Rhoving/iac-repo#319

claude-bot referenced this issue

2026-08-17 19:35:13 +00:00

deploy: nothing syncs deploy/compose.yaml to the hosts — both are a month stale, and #117's worker split isn't deployed anywhere #155

rbrooks referenced this issue from a commit

2026-08-17 19:42:18 +00:00

fix(worker): give the worker container a liveness probe it can actually pass

rbrooks closed this issue

2026-08-17 19:42:18 +00:00

rbrooks referenced this issue from a commit

2026-08-17 19:42:39 +00:00

fix(tealeaves): point the worker healthcheck at the worker's own probe

claude-bot commented

2026-08-17 19:45:57 +00:00

Author

Contributor

Auto-closed by 2710a9f. Reopening — the app-side half is done and verified, but the container is still unhealthy on both hosts, which is what this issue is actually about.

Verified working on dev

Dev's Watchtower picked up the new :beta within ~2 minutes. The heartbeat and probe both do what they should:

$ docker exec tealeaves-redis-1 redis-cli GET tealeaves:worker:heartbeat
1786995927513
$ docker exec tealeaves-redis-1 redis-cli TTL tealeaves:worker:heartbeat
39                       # refreshing on schedule — 15s interval against a 45s TTL

$ docker exec tealeaves-worker-1 node dist/healthcheckWorker.js; echo $?
0

Chose option 2 from the list above, with the Redis variant rather than an HTTP listener: no new port, and it tests the thing the container is actually for. A file-on-disk heartbeat was the tempting shortcut and is the wrong shape — it would keep ticking from a timer while Redis was unreachable and nothing was being consumed, reporting healthy for precisely the failure worth catching.

Still outstanding

The container reports unhealthy regardless, because Docker is still running the inherited probe — the compose-side override is Rhoving/iac-repo#325, open and not yet merged.

host	pin	state after #325 merges + play runs
ai-dev-server	`:beta`	fixed — already has the probe
docker-host	`9.4.0`	still unhealthy: that tag predates the probe

Prod needs a release tag containing 2710a9f and its tealeaves_{api,app}_image pin bumped before it clears. Not a regression in the meantime — the probe fails file-not-found instead of connection-refused, and it was already red.

Closing this again once both hosts show healthy.

Auto-closed by `2710a9f`. **Reopening** — the app-side half is done and verified, but the container is still `unhealthy` on both hosts, which is what this issue is actually about. ## Verified working on dev Dev's Watchtower picked up the new `:beta` within ~2 minutes. The heartbeat and probe both do what they should: ``` $ docker exec tealeaves-redis-1 redis-cli GET tealeaves:worker:heartbeat 1786995927513 $ docker exec tealeaves-redis-1 redis-cli TTL tealeaves:worker:heartbeat 39 # refreshing on schedule — 15s interval against a 45s TTL $ docker exec tealeaves-worker-1 node dist/healthcheckWorker.js; echo $? 0 ``` Chose **option 2** from the list above, with the Redis variant rather than an HTTP listener: no new port, and it tests the thing the container is actually for. A file-on-disk heartbeat was the tempting shortcut and is the wrong shape — it would keep ticking from a timer while Redis was unreachable and nothing was being consumed, reporting healthy for precisely the failure worth catching. ## Still outstanding The container reports `unhealthy` regardless, because Docker is still running the **inherited** probe — the compose-side override is `Rhoving/iac-repo#325`, open and not yet merged. | host | pin | state after #325 merges + play runs | |---|---|---| | ai-dev-server | `:beta` | **fixed** — already has the probe | | docker-host | `9.4.0` | still unhealthy: that tag predates the probe | Prod needs a release tag containing `2710a9f` and its `tealeaves_{api,app}_image` pin bumped before it clears. Not a regression in the meantime — the probe fails file-not-found instead of connection-refused, and it was already red. Closing this again once both hosts show `healthy`.

claude-bot reopened this issue

2026-08-17 19:45:59 +00:00