Backup spawns (restic/pg_dump) have no timeout - a hung repo silently stops all backups #86

New issue

Closed

opened 2026-07-15 19:49:57 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-15 19:49:57 +00:00

Contributor

Severity: Medium · Confidence: High · Effort: S · Category: ops

Evidence

api/src/services/backup.ts:94-155 - run/runBuffer/spawnPgDump have no timeout, no signal.
api/src/services/backup.ts:452 - backup-schedule worker concurrency: 1.

Problem
A stuck restic (unreachable B2/S3/SFTP, network stall) or pg_dump never resolves. With concurrency 1 and no timeout, the single backup worker blocks permanently; every subsequent scheduled backup never runs while the job sits in status='running' forever.

Impact
Silent failure of the backup promise - the dashboard shows "running", not "failed", so no alert fires. Backups quietly stop.

Fix
Wrap spawns with a timeout (kill child + reject after N minutes) and, on the next scheduler tick, treat a job running longer than a threshold as failed (fire the alert).

Acceptance criteria

A restic/pg_dump exceeding the timeout is killed and the job marked failed.
A stale running job is reconciled to failed and fires the configured alert.

Related: F-07 (restore), F-08 (shutdown).

Filed from the 2026-07-15 codebase audit. Full report: docs/.internal/report-2026-07-15.md (gitignored).

**Severity:** Medium · **Confidence:** High · **Effort:** S · Category: ops **Evidence** - `api/src/services/backup.ts:94-155` - `run`/`runBuffer`/`spawnPgDump` have no timeout, no `signal`. - `api/src/services/backup.ts:452` - backup-schedule worker `concurrency: 1`. **Problem** A stuck `restic` (unreachable B2/S3/SFTP, network stall) or `pg_dump` never resolves. With concurrency 1 and no timeout, the single backup worker blocks permanently; every subsequent scheduled backup never runs while the job sits in `status='running'` forever. **Impact** Silent failure of the backup promise - the dashboard shows "running", not "failed", so no alert fires. Backups quietly stop. **Fix** Wrap spawns with a timeout (kill child + reject after N minutes) and, on the next scheduler tick, treat a job `running` longer than a threshold as failed (fire the alert). **Acceptance criteria** - [ ] A restic/pg_dump exceeding the timeout is killed and the job marked failed. - [ ] A stale `running` job is reconciled to failed and fires the configured alert. Related: F-07 (restore), F-08 (shutdown). --- _Filed from the 2026-07-15 codebase audit. Full report: `docs/.internal/report-2026-07-15.md` (gitignored)._

claude-bot added this to the v7.1.0 - Core hardening & security milestone

2026-07-15 19:49:57 +00:00

claude-bot added the

bug

label

2026-07-15 19:49:57 +00:00

rbrooks referenced this issue from a commit

2026-07-16 21:34:18 +00:00

fix(v7.1): wave 3 — backup recovery & process lifecycle (#87, #84, #85, #86)

claude-bot referenced this issue

2026-07-16 21:47:32 +00:00

Backup entry-restore is broken - INSERT omits NOT NULL project_id and uses invalid certainty #84

claude-bot referenced this issue

2026-07-16 22:03:52 +00:00

Add graceful shutdown - close BullMQ workers, HTTP server, and pools on SIGTERM #85

claude-bot commented

2026-07-16 22:03:59 +00:00

Author

Contributor

Fixed in b640072 (v7.1.0 wave 3).

Spawn timeouts: run, runBuffer, runWithStdinBuffer and spawnPgDump now arm a kill-timer (SPAWN_TIMEOUT_MS, 2 h) via a shared armTimeout() helper — the child is SIGKILLed and the promise rejects, so the job is failed rather than hanging the concurrency-1 worker forever.
Also fixed: none of these had an error handler, so a spawn failure (e.g. restic/pg_dump binary missing) left the promise pending forever — a second silent-hang path the issue didn't name. All four now reject on 'error'.
Stale reconciliation: new reconcileStaleJobs() fails any job still running past STALE_JOB_MS (3 h — deliberately > the spawn timeout, so a timed-out spawn fails the job first and this only catches jobs orphaned by a hard crash), sets a descriptive error_message, and fires the destination's alert_webhook_url. It runs on each backup-worker tick (before the run, so a stale row can't block the tick) and errors are caught so it can't break the backup itself.

Threshold values are parameterised (make_interval(secs => $1)), per the project's SQL invariant.

Acceptance criteria:

A restic/pg_dump exceeding the timeout is killed and the job marked failed.
A stale running job is reconciled to failed and fires the configured alert.

CI green (238/238 tests). Related: #85 (graceful shutdown) prevents the common cause of a stranded running job in the first place.

Fixed in `b640072` (v7.1.0 wave 3). - **Spawn timeouts:** `run`, `runBuffer`, `runWithStdinBuffer` and `spawnPgDump` now arm a kill-timer (`SPAWN_TIMEOUT_MS`, 2 h) via a shared `armTimeout()` helper — the child is `SIGKILL`ed and the promise rejects, so the job is failed rather than hanging the concurrency-1 worker forever. - **Also fixed:** none of these had an `error` handler, so a spawn failure (e.g. `restic`/`pg_dump` binary missing) left the promise **pending forever** — a second silent-hang path the issue didn't name. All four now `reject` on `'error'`. - **Stale reconciliation:** new `reconcileStaleJobs()` fails any job still `running` past `STALE_JOB_MS` (3 h — deliberately > the spawn timeout, so a timed-out spawn fails the job first and this only catches jobs orphaned by a hard crash), sets a descriptive `error_message`, and fires the destination's `alert_webhook_url`. It runs on each backup-worker tick (before the run, so a stale row can't block the tick) and errors are caught so it can't break the backup itself. Threshold values are parameterised (`make_interval(secs => $1)`), per the project's SQL invariant. **Acceptance criteria:** - [x] A restic/pg_dump exceeding the timeout is killed and the job marked failed. - [x] A stale `running` job is reconciled to failed and fires the configured alert. CI green (238/238 tests). Related: #85 (graceful shutdown) prevents the common cause of a stranded `running` job in the first place.

claude-bot closed this issue

2026-07-16 22:04:37 +00:00

claude-bot referenced this issue

2026-07-16 22:05:06 +00:00

Roadmap: sequence a core-hardening pass before v8 and test the v7 federation trust boundary #92