[Ops] Add failure counters and alerting hooks for background jobs #86

New issue

Closed

opened 2026-07-14 19:46:49 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-07-14 19:46:49 +00:00

Contributor

Context

Observability is decent if someone is watching: structured JSON logs, a token-gated Prometheus /metrics endpoint (webapp/backend/app/metrics.py), and /ready. But background-job failures only surface as logger.error to stdout — e.g. run_scheduled_backup failed at webapp/backend/app/tasks/reminder_tasks.py:1926, and similar except-paths throughout the task module.

Current behavior

A transcription or backup failure at 2am is a log line nobody reads. There is no counter to alert on and no error tracker — the first sign of a broken backup job could be needing a backup that isn't there.

Fix / Spec

Add Prometheus counters to webapp/backend/app/metrics.py:
- backup_failed_total
- celery_task_failed_total{task_name} — implement via a Celery task_failure signal handler registered in the Celery app module, so every task is covered without touching each one
- bot_event_delivery_failed_total
Increment them in the relevant except paths (backup: reminder_tasks.py:1926 area; bot delivery: the notify/publish failure paths). The task_failure signal covers the general case.
Document 2-3 example Prometheus alert rules in docs/OPERATIONS.md:
- backup has failed (or not succeeded) in the last 25h
- rate(celery_task_failed_total[15m]) > 0 sustained for 15m
- bot event delivery failures increasing
Optional: SENTRY_DSN env hook — sentry-sdk init in main.py plus the Celery integration, entirely a no-op when the env var is unset. Do not make sentry-sdk a hard runtime requirement of any code path when unset.

Acceptance criteria

Forcing a backup failure (e.g. bad pg_dump path in a dev stack) increments backup_failed_total, visible at /metrics.
A deliberately failing Celery task increments celery_task_failed_total with its task name.
Example alert rules are in docs/OPERATIONS.md.
App boots and runs unchanged with no SENTRY_DSN set.

References

webapp/backend/app/metrics.py
webapp/backend/app/tasks/reminder_tasks.py:1926 (backup failure path)

Filed from the July 2026 full-project review.

## Context Observability is decent *if someone is watching*: structured JSON logs, a token-gated Prometheus `/metrics` endpoint (`webapp/backend/app/metrics.py`), and `/ready`. But background-job failures only surface as `logger.error` to stdout — e.g. `run_scheduled_backup failed` at `webapp/backend/app/tasks/reminder_tasks.py:1926`, and similar except-paths throughout the task module. ## Current behavior A transcription or backup failure at 2am is a log line nobody reads. There is no counter to alert on and no error tracker — the first sign of a broken backup job could be needing a backup that isn't there. ## Fix / Spec 1. Add Prometheus counters to `webapp/backend/app/metrics.py`: - `backup_failed_total` - `celery_task_failed_total{task_name}` — implement via a Celery `task_failure` signal handler registered in the Celery app module, so every task is covered without touching each one - `bot_event_delivery_failed_total` 2. Increment them in the relevant `except` paths (backup: `reminder_tasks.py:1926` area; bot delivery: the notify/publish failure paths). The `task_failure` signal covers the general case. 3. Document 2-3 example Prometheus alert rules in `docs/OPERATIONS.md`: - backup has failed (or not succeeded) in the last 25h - `rate(celery_task_failed_total[15m]) > 0` sustained for 15m - bot event delivery failures increasing 4. Optional: `SENTRY_DSN` env hook — `sentry-sdk` init in `main.py` plus the Celery integration, entirely a no-op when the env var is unset. Do not make sentry-sdk a hard runtime requirement of any code path when unset. ## Acceptance criteria - [ ] Forcing a backup failure (e.g. bad pg_dump path in a dev stack) increments `backup_failed_total`, visible at `/metrics`. - [ ] A deliberately failing Celery task increments `celery_task_failed_total` with its task name. - [ ] Example alert rules are in `docs/OPERATIONS.md`. - [ ] App boots and runs unchanged with no `SENTRY_DSN` set. ## References - `webapp/backend/app/metrics.py` - `webapp/backend/app/tasks/reminder_tasks.py:1926` (backup failure path) _Filed from the July 2026 full-project review._

claude-bot added the

backend

enhancement

labels

2026-07-14 19:46:57 +00:00

claude-bot added this to the v3.4.0 - Reliability, Operations & Self-Hosting milestone

2026-07-14 19:46:58 +00:00