feat(ops): failure counters + alerting hooks for background jobs (#86) #171

Merged

claude-bot merged 1 commit from feat/86-failure-counters into main

2026-07-16 01:38:15 +00:00

claude-bot commented

2026-07-16 01:36:06 +00:00

Contributor

Summary

Background-job failures only surfaced as logger.error to stdout — a backup/transcription failure at 2am was a log line nobody read, with no counter to alert on. Adds Prometheus counters, example alert rules, and an optional Sentry hook.

Changes

metrics.py: backup_failed_total, celery_task_failed_total{task_name}, bot_event_delivery_failed_total.
celery_app.py: a @task_failure.connect handler increments celery_task_failed_total{task_name} for every task automatically (no per-task edits).
reminder_tasks.py: backup_failed_total in the run_scheduled_backup except path; bot_event_delivery_failed_total in the delivery-failure except paths. Only .inc() added — no delivery behaviour changed. For the two-tier HTTP→stream paths it counts only the terminal failure (both mechanisms exhausted), not an inner HTTP failure that then succeeds via fallback.
Optional Sentry: sentry_dsn config (default empty); main.py inits sentry-sdk (FastAPI + Celery integrations) only when set — the import itself is skipped when unset, so it's a complete no-op. sentry-sdk pinned in requirements.
OPERATIONS.md: new "Alerting" subsection with 3 example Prometheus rules (backup staleness >25h; rate(celery_task_failed_total[15m]) > 0 for 15m; bot-delivery failures increasing).

Verification

Backend full suite 394 passed (+ new tests: signal handler increments the labelled counter incl. a real task_failure.send(); backup except-path increments; /metrics exposes all three). Confirmed the suite passes with sentry-sdk not importable (unset-DSN path fully guarded). ruff 0.4.4 clean.

Closes #86

🤖 Generated with Claude Code

## Summary Background-job failures only surfaced as `logger.error` to stdout — a backup/transcription failure at 2am was a log line nobody read, with no counter to alert on. Adds Prometheus counters, example alert rules, and an optional Sentry hook. ## Changes - **`metrics.py`**: `backup_failed_total`, `celery_task_failed_total{task_name}`, `bot_event_delivery_failed_total`. - **`celery_app.py`**: a `@task_failure.connect` handler increments `celery_task_failed_total{task_name}` for **every** task automatically (no per-task edits). - **`reminder_tasks.py`**: `backup_failed_total` in the `run_scheduled_backup` except path; `bot_event_delivery_failed_total` in the delivery-failure except paths. Only `.inc()` added — no delivery behaviour changed. For the two-tier HTTP→stream paths it counts only the **terminal** failure (both mechanisms exhausted), not an inner HTTP failure that then succeeds via fallback. - **Optional Sentry**: `sentry_dsn` config (default empty); `main.py` inits `sentry-sdk` (FastAPI + Celery integrations) **only when set** — the `import` itself is skipped when unset, so it's a complete no-op. `sentry-sdk` pinned in requirements. - **`OPERATIONS.md`**: new "Alerting" subsection with 3 example Prometheus rules (backup staleness >25h; `rate(celery_task_failed_total[15m]) > 0` for 15m; bot-delivery failures increasing). ## Verification Backend full suite **394 passed** (+ new tests: signal handler increments the labelled counter incl. a real `task_failure.send()`; backup except-path increments; `/metrics` exposes all three). Confirmed the suite passes with `sentry-sdk` **not importable** (unset-DSN path fully guarded). ruff 0.4.4 clean. Closes #86 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit

2026-07-16 01:36:07 +00:00

feat(observability): add failure counters and alerting hooks for background jobs

CI / Backend lint (ruff) (pull_request) Successful in 1m9s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 1m41s

Details

CI / Bot tests and audit (pull_request) Successful in 2m40s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 4m25s

Details

CI / Docker image build (pull_request) Successful in 8m59s

Details

e73f0087ab

Background-job failures previously only surfaced as logger.error to stdout.
Add three Prometheus counters (backup_failed_total, celery_task_failed_total
labelled by task_name, bot_event_delivery_failed_total), wire them into the
existing failure paths without changing delivery behaviour, register a
Celery task_failure signal handler so every task is covered automatically,
document example alert rules in docs/OPERATIONS.md, and add an optional
Sentry hook (sentry-sdk, guarded behind an empty-by-default SENTRY_DSN so the
app boots and runs identically, and sentry-sdk need not be installed, when
unset).

Closes #86

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed

2026-07-16 01:37:08 +00:00

claude-bot merged commit f520ad5a37 into main

2026-07-16 01:38:15 +00:00

claude-bot deleted branch feat/86-failure-counters

2026-07-16 01:38:15 +00:00

claude-bot referenced this pull request from a commit

2026-07-16 01:38:16 +00:00

Merge pull request 'feat(ops): failure counters + alerting hooks for background jobs (#86)' (#171) from feat/86-failure-counters into main