[Ops] Add worker/beat healthchecks, container resource limits, and Redis AOF persistence #84

New issue

Closed

opened 2026-07-14 19:46:28 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-07-14 19:46:28 +00:00

Contributor

Context

docker-compose.yml defines healthchecks for db (:21), redis (:32), backend (:55), and bot (:124) — but none for worker (:63-78), beat (:80-94), or frontend (:96-107). No service has mem_limit/cpus anywhere. redis:7-alpine (:29) runs with default RDB-snapshot-only persistence while holding user sessions (DB 0), the Celery broker (DB 1) with scheduled tasks queued up to 7 days out, and results (DB 2). The bot's depends_on gates backend only on service_started (:117-119).

Current behavior

A hung/dead Celery worker or beat looks identical to a healthy one in docker compose ps — reminders and transcription silently stop.
A Redis crash between RDB snapshots drops queued Celery work (lost reminders) and logs every user out.
A runaway container (e.g. worker during transcription post-processing) can take down the whole host — nothing bounds memory.
The bot can start before the backend is actually serving.

Fix / Spec

Worker healthcheck: celery -A app.tasks.celery_app inspect ping -d celery@$$HOSTNAME (or equivalent that pings only the local worker), sensible interval/timeout/retries/start_period.
Beat healthcheck: heartbeat-file pattern — beat's schedule loop (or a tiny wrapper) touches a file; healthcheck fails if the file is older than N minutes.
Memory limits on worker, beat, backend, and bot — conservative values consistent with a 4 GB host (e.g. worker 1g, backend 512m, beat 256m, bot 512m — tune as measured), with a comment/doc note on how to raise them.
Redis AOF: add --appendonly yes to the redis command; add a note in docs/OPERATIONS.md on the durability-vs-IO trade-off (appendfsync everysec default is fine).
Bot ordering: bump depends_on.backend from service_started to service_healthy (docker-compose.yml:117-119).
Optional: frontend healthcheck (wget/curl of /).
Verify docker-compose.prod.yml composes cleanly with these additions.

Acceptance criteria

docker compose ps shows health status for worker and beat.
docker pause/SIGSTOP on the worker flips it to unhealthy within a few intervals.
Stack restarts cleanly with AOF enabled; sessions survive a docker compose restart redis.
Memory limits present on worker/beat/backend/bot; documented how to adjust.

References

docker-compose.yml:21, :32, :55, :124 (existing healthchecks)
docker-compose.yml:63-78 (worker), :80-94 (beat), :96-107 (frontend), :28-37 (redis), :117-119 (bot depends_on)

Filed from the July 2026 full-project review.

## Context `docker-compose.yml` defines healthchecks for db (`:21`), redis (`:32`), backend (`:55`), and bot (`:124`) — but none for **worker** (`:63-78`), **beat** (`:80-94`), or **frontend** (`:96-107`). No service has `mem_limit`/`cpus` anywhere. `redis:7-alpine` (`:29`) runs with default RDB-snapshot-only persistence while holding user sessions (DB 0), the Celery broker (DB 1) with scheduled tasks queued up to 7 days out, and results (DB 2). The bot's `depends_on` gates backend only on `service_started` (`:117-119`). ## Current behavior - A hung/dead Celery worker or beat looks identical to a healthy one in `docker compose ps` — reminders and transcription silently stop. - A Redis crash between RDB snapshots drops queued Celery work (lost reminders) and logs every user out. - A runaway container (e.g. worker during transcription post-processing) can take down the whole host — nothing bounds memory. - The bot can start before the backend is actually serving. ## Fix / Spec 1. **Worker healthcheck**: `celery -A app.tasks.celery_app inspect ping -d celery@$$HOSTNAME` (or equivalent that pings only the local worker), sensible interval/timeout/retries/start_period. 2. **Beat healthcheck**: heartbeat-file pattern — beat's schedule loop (or a tiny wrapper) touches a file; healthcheck fails if the file is older than N minutes. 3. **Memory limits** on worker, beat, backend, and bot — conservative values consistent with a 4 GB host (e.g. worker 1g, backend 512m, beat 256m, bot 512m — tune as measured), with a comment/doc note on how to raise them. 4. **Redis AOF**: add `--appendonly yes` to the redis command; add a note in `docs/OPERATIONS.md` on the durability-vs-IO trade-off (`appendfsync everysec` default is fine). 5. **Bot ordering**: bump `depends_on.backend` from `service_started` to `service_healthy` (`docker-compose.yml:117-119`). 6. Optional: frontend healthcheck (wget/curl of `/`). 7. Verify `docker-compose.prod.yml` composes cleanly with these additions. ## Acceptance criteria - [ ] `docker compose ps` shows health status for worker and beat. - [ ] `docker pause`/SIGSTOP on the worker flips it to unhealthy within a few intervals. - [ ] Stack restarts cleanly with AOF enabled; sessions survive a `docker compose restart redis`. - [ ] Memory limits present on worker/beat/backend/bot; documented how to adjust. ## References - `docker-compose.yml:21`, `:32`, `:55`, `:124` (existing healthchecks) - `docker-compose.yml:63-78` (worker), `:80-94` (beat), `:96-107` (frontend), `:28-37` (redis), `:117-119` (bot depends_on) _Filed from the July 2026 full-project review._

claude-bot added the

backend

enhancement

labels

2026-07-14 19:46:38 +00:00

claude-bot added this to the v3.4.0 - Reliability, Operations & Self-Hosting milestone

2026-07-14 19:46:39 +00:00