rbrooks/TeaLeaves

Watch

Fork 0

Add graceful shutdown - close BullMQ workers, HTTP server, and pools on SIGTERM #85

New issue

Closed

opened 2026-07-15 19:49:56 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-15 19:49:56 +00:00

Contributor

Severity: Medium · Confidence: High · Effort: S · Category: ops

Evidence

api/src/index.ts - no SIGTERM/SIGINT handler; workers started at :17-26 are never closed; app.listen never closed.
api/src/services/mediaProcessorWorker.ts:23-51 (ffmpeg/ffprobe), api/src/services/backup.ts:96,111,143 (restic/pg_dump).

Problem
On the documented deploy (docker compose up --build, CLAUDE.md step 6) the container gets SIGTERM. Nothing calls worker.close() or closes the server/pools, so in-flight import/media/backup jobs are hard-killed, BullMQ locks linger, and ffmpeg/restic/pg_dump children can orphan.

Impact
Interrupted backups/imports on every deploy; possible zombie children; a backup left stuck status='running' which then blocks the /trigger guard (backup.ts route :44-49).

Fix
Return worker refs from the startX() functions; on SIGTERM/SIGINT await Promise.all(workers.map(w => w.close())), close the HTTP server and pg/redis pools, then exit.

Acceptance criteria

SIGTERM drains in-flight jobs (bounded) before exit.
No backup job is left in status='running' after a normal redeploy.

Related: F-09 (backup timeouts), F-13 (migration ordering).

Filed from the 2026-07-15 codebase audit. Full report: docs/.internal/report-2026-07-15.md (gitignored).

**Severity:** Medium · **Confidence:** High · **Effort:** S · Category: ops **Evidence** - `api/src/index.ts` - no SIGTERM/SIGINT handler; workers started at :17-26 are never closed; `app.listen` never closed. - `api/src/services/mediaProcessorWorker.ts:23-51` (ffmpeg/ffprobe), `api/src/services/backup.ts:96,111,143` (restic/pg_dump). **Problem** On the documented deploy (`docker compose up --build`, CLAUDE.md step 6) the container gets SIGTERM. Nothing calls `worker.close()` or closes the server/pools, so in-flight import/media/backup jobs are hard-killed, BullMQ locks linger, and ffmpeg/restic/pg_dump children can orphan. **Impact** Interrupted backups/imports on every deploy; possible zombie children; a backup left stuck `status='running'` which then blocks the `/trigger` guard (backup.ts route :44-49). **Fix** Return worker refs from the `startX()` functions; on SIGTERM/SIGINT `await Promise.all(workers.map(w => w.close()))`, close the HTTP server and pg/redis pools, then exit. **Acceptance criteria** - [ ] SIGTERM drains in-flight jobs (bounded) before exit. - [ ] No backup job is left in `status='running'` after a normal redeploy. Related: F-09 (backup timeouts), F-13 (migration ordering). --- _Filed from the 2026-07-15 codebase audit. Full report: `docs/.internal/report-2026-07-15.md` (gitignored)._

claude-bot added this to the v7.1.0 - Core hardening & security milestone

2026-07-15 19:49:56 +00:00

claude-bot added the

bug

label

2026-07-15 19:49:56 +00:00

rbrooks referenced this issue from a commit

2026-07-16 21:34:18 +00:00

fix(v7.1): wave 3 — backup recovery & process lifecycle (#87, #84, #85, #86)

rbrooks referenced this issue from a commit

2026-07-16 21:38:06 +00:00

fix(v7.1): make SIGTERM actually reach node so shutdown runs (#85)

claude-bot referenced this issue

2026-07-16 21:47:25 +00:00

Run migrations before starting workers and the backup scheduler #87

claude-bot commented

2026-07-16 22:03:52 +00:00

Author

Contributor

Fixed in b640072 + eb07b31 (v7.1.0 wave 3).

Every startX() now returns its Worker; api/src/index.ts collects them and on SIGTERM/SIGINT: closes the HTTP server (stop accepting new requests) → await Promise.all(workers.map(w => w.close())) (each worker finishes its current job) → closes the Redis session client and the pg pool → exit 0. A 30 s SHUTDOWN_TIMEOUT_MS hard cap forces exit rather than hanging.

⚠️ The handler alone was inert — worth knowing. On first test, docker compose stop api still died with npm error signal SIGTERM and exit code 1, and none of the [shutdown] logs appeared. The image ran CMD ["npm", "start"], and npm does not forward SIGTERM to its node child — so the handler never ran. Fixed in eb07b31:

api/Dockerfile execs node dist/index.js directly (node is the signal target).
docker-compose.yml sets init: true — tini as PID 1, which forwards the signal and reaps the orphaned restic/pg_dump/ffmpeg children this issue flags — plus stop_grace_period: 40s to allow the drain.

Acceptance criteria (verified live on the dev server):

SIGTERM drains in-flight jobs (bounded) before exit — PID 1 is /sbin/docker-init, and docker compose stop api now logs [shutdown] SIGTERM received — draining in-flight work… → [shutdown] clean exit, exit code 0 (was 1).
No backup job is left in status='running' after a normal redeploy — workers drain cleanly, and reconcileStaleJobs() (#86) is the backstop for a hard crash.

CI green (238/238 tests).

Fixed in `b640072` + `eb07b31` (v7.1.0 wave 3). Every `startX()` now returns its `Worker`; `api/src/index.ts` collects them and on SIGTERM/SIGINT: closes the HTTP server (stop accepting new requests) → `await Promise.all(workers.map(w => w.close()))` (each worker finishes its current job) → closes the Redis session client and the pg pool → exit 0. A 30 s `SHUTDOWN_TIMEOUT_MS` hard cap forces exit rather than hanging. **⚠️ The handler alone was inert — worth knowing.** On first test, `docker compose stop api` still died with `npm error signal SIGTERM` and **exit code 1**, and none of the `[shutdown]` logs appeared. The image ran `CMD ["npm", "start"]`, and npm does not forward SIGTERM to its node child — so the handler never ran. Fixed in `eb07b31`: - `api/Dockerfile` execs `node dist/index.js` directly (node is the signal target). - `docker-compose.yml` sets `init: true` — tini as PID 1, which forwards the signal *and* reaps the orphaned restic/pg_dump/ffmpeg children this issue flags — plus `stop_grace_period: 40s` to allow the drain. **Acceptance criteria** (verified live on the dev server): - [x] SIGTERM drains in-flight jobs (bounded) before exit — PID 1 is `/sbin/docker-init`, and `docker compose stop api` now logs `[shutdown] SIGTERM received — draining in-flight work…` → `[shutdown] clean exit`, **exit code 0** (was 1). - [x] No backup job is left in `status='running'` after a normal redeploy — workers drain cleanly, and `reconcileStaleJobs()` (#86) is the backstop for a hard crash. CI green (238/238 tests).

claude-bot referenced this issue

2026-07-16 22:03:59 +00:00

Backup spawns (restic/pg_dump) have no timeout - a hung repo silently stops all backups #86

claude-bot closed this issue

2026-07-16 22:04:33 +00:00

claude-bot referenced this issue

2026-07-16 22:05:06 +00:00

Roadmap: sequence a core-hardening pass before v8 and test the v7 federation trust boundary #92

rbrooks referenced this issue from a commit

2026-07-17 02:36:46 +00:00

fix(v7.2): wave 2 — ops hygiene (#95, #100, #101, #102)

claude-bot referenced this issue