feat(ops): worker/beat healthchecks, resource limits, Redis AOF (#84) #169

Merged

claude-bot merged 2 commits from feat/84-healthchecks-limits into main

2026-07-16 00:37:09 +00:00

claude-bot commented

2026-07-16 00:33:25 +00:00

Contributor

Summary

docker-compose.yml had healthchecks for db/redis/backend/bot but none for worker/beat/frontend, no memory limits anywhere, and redis ran RDB-only persistence while holding sessions (DB 0) + the Celery broker (DB 1, tasks queued up to 7 days out). A hung worker/beat looked healthy; a redis crash between snapshots dropped queued work and logged everyone out; a runaway container could take down the host.

Changes (`docker-compose.yml`)

Worker healthcheck: celery … inspect ping -d celery@$$HOSTNAME (pings only the local worker; generous start_period).
Beat healthcheck: a background loop touches /tmp/beat-heartbeat every 60s (wrapping beat via sh -c … exec), healthcheck fails if the file is >3min stale — the celerybeat-schedule mtime isn't a reliable liveness signal.
Memory limits (Compose v2 top-level mem_limit): worker 1g, backend 512m, beat 256m, bot 512m; how to raise them is documented.
Redis AOF: --appendonly yes + an OPERATIONS.md note on the appendfsync everysec durability/IO trade-off.
Bot ordering: depends_on.backend service_started → service_healthy.
Frontend healthcheck: wget --spider http://localhost/ (verified the current nginx:alpine ships wget).

Preserves #89's one-shot migrate service and the migrate: service_completed_successfully ordering (merged main in and reconciled the beat block so it has both the migrate dependency and the heartbeat command).

Verification

docker compose config for base, +dev, and +prod overlays all resolve cleanly with the new healthchecks + mem_limits (validated on the dev Docker host). Runtime acceptance (pause worker → unhealthy; sessions survive docker compose restart redis with AOF) is documented as a manual checklist in OPERATIONS.md.

Closes #84

🤖 Generated with Claude Code

## Summary `docker-compose.yml` had healthchecks for db/redis/backend/bot but none for worker/beat/frontend, no memory limits anywhere, and redis ran RDB-only persistence while holding sessions (DB 0) + the Celery broker (DB 1, tasks queued up to 7 days out). A hung worker/beat looked healthy; a redis crash between snapshots dropped queued work and logged everyone out; a runaway container could take down the host. ## Changes (`docker-compose.yml`) - **Worker healthcheck**: `celery … inspect ping -d celery@$$HOSTNAME` (pings only the local worker; generous start_period). - **Beat healthcheck**: a background loop `touch`es `/tmp/beat-heartbeat` every 60s (wrapping beat via `sh -c … exec`), healthcheck fails if the file is >3min stale — the `celerybeat-schedule` mtime isn't a reliable liveness signal. - **Memory limits** (Compose v2 top-level `mem_limit`): worker 1g, backend 512m, beat 256m, bot 512m; how to raise them is documented. - **Redis AOF**: `--appendonly yes` + an OPERATIONS.md note on the `appendfsync everysec` durability/IO trade-off. - **Bot ordering**: `depends_on.backend` `service_started` → `service_healthy`. - **Frontend healthcheck**: `wget --spider http://localhost/` (verified the current nginx:alpine ships wget). Preserves #89's one-shot `migrate` service and the `migrate: service_completed_successfully` ordering (merged main in and reconciled the beat block so it has both the migrate dependency and the heartbeat command). ## Verification `docker compose config` for base, `+dev`, and `+prod` overlays all resolve cleanly with the new healthchecks + mem_limits (validated on the dev Docker host). Runtime acceptance (pause worker → unhealthy; sessions survive `docker compose restart redis` with AOF) is documented as a manual checklist in OPERATIONS.md. Closes #84 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 2 commits

2026-07-16 00:33:25 +00:00

feat(compose): add worker/beat/frontend healthchecks, mem limits, redis AOF ed3b9a9c36

Worker gets a celery inspect ping healthcheck scoped to its own hostname;
beat gets a background heartbeat-file touch loop plus a freshness check
(celerybeat-schedule's mtime only changes on schedule edits, not per tick,
so it can't be used directly); frontend gets a wget --spider check (verified
nginx:alpine ships both wget and curl). Add mem_limit to worker/backend/
beat/bot sized for a ~4GB host, enable redis --appendonly yes since redis
holds sessions and a Celery broker queued up to 7 days out, and tighten the
bot's dependency on backend to service_healthy instead of service_started.
Document the new limits, the AOF trade-off, and a manual verification
checklist in docs/OPERATIONS.md.

Closes #84

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into feat/84-healthchecks-limits

CI / Backend migration, tests, and audit (pull_request) Failing after 2s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 1m20s

Details

CI / Docker image build (pull_request) Successful in 12s

Details

CI / Backend lint (ruff) (pull_request) Successful in 36s

Details

CI / Bot tests and audit (pull_request) Successful in 1m49s

Details

aa99163f1c