rbrooks/WeatherBot

Watch

Fork 0

Add data-retention/cleanup jobs — most tables and the SPC image cache grow forever #52

New issue

Closed

opened 2026-07-18 05:31:18 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-18 05:31:18 +00:00

Contributor

Only sent_forecasts and product_freshness_samples have delete-based
retention. Growing without bound: sent_alerts (MCD rows never even clear —
expires=None, spc_processor.py:568-577, excluded by the expiry job's
expires != None filter), nws_alert_snapshots, ai_summary_records +
ai_summary_attempts (large JSONB/source-text payloads),
spc_outlook_pages/_impact_locations/_risk_transitions (retained_until
only gates reads), public_tokens (disabled, never deleted), and the
date-keyed SPC PNG/GIF disk cache (radar.py:284-311 — new files daily, no
pruning).

Scope: (1) set a real expires on MCD SentAlert rows; (2) prune SPC cache
files older than N days; (3) a scheduled retention job with per-table,
config-backed windows — deliberately long defaults, chosen with #22 in mind
(the Historical Explorer will consume this history; deleting aggressively now
would gut it). Document the windows in README.

Acceptance criteria:

MCD rows expire and clear like other SPC rows
radar_cache/ no longer accumulates unbounded date-keyed files
Retention job covers the listed tables with configurable windows
Windows documented; #22 references the retention design

Filed from the 2026-07-17 codebase audit (docs/.internal/report-2026-07-17.md), finding F-13.

Only `sent_forecasts` and `product_freshness_samples` have delete-based retention. Growing without bound: `sent_alerts` (MCD rows never even clear — `expires=None`, spc_processor.py:568-577, excluded by the expiry job's `expires != None` filter), `nws_alert_snapshots`, `ai_summary_records` + `ai_summary_attempts` (large JSONB/source-text payloads), `spc_outlook_pages`/`_impact_locations`/`_risk_transitions` (`retained_until` only gates reads), `public_tokens` (disabled, never deleted), and the date-keyed SPC PNG/GIF disk cache (radar.py:284-311 — new files daily, no pruning). Scope: (1) set a real `expires` on MCD SentAlert rows; (2) prune SPC cache files older than N days; (3) a scheduled retention job with per-table, config-backed windows — deliberately long defaults, chosen with #22 in mind (the Historical Explorer will consume this history; deleting aggressively now would gut it). Document the windows in README. **Acceptance criteria:** - [ ] MCD rows expire and clear like other SPC rows - [ ] radar_cache/ no longer accumulates unbounded date-keyed files - [ ] Retention job covers the listed tables with configurable windows - [ ] Windows documented; #22 references the retention design --- _Filed from the 2026-07-17 codebase audit (`docs/.internal/report-2026-07-17.md`), finding F-13._

claude-bot added this to the v1.1.0 - Operations, observability & CD milestone

2026-07-18 05:31:18 +00:00

claude-bot added the

type:maintenance

area:observability

labels

2026-07-18 05:31:18 +00:00

rbrooks referenced this issue from a commit

2026-07-18 18:07:23 +00:00

Add public-link console, /p/* rate limiting, SMS links, retention, config I/O

claude-bot referenced this issue

2026-07-18 18:07:41 +00:00

v1.1.0 Phase 4: public-link console, rate limiting, retention, config I/O #97

claude-bot commented

2026-07-18 18:17:53 +00:00

Author

Contributor

Done in #97 (merged). MCD SentAlert rows now get a real 6h expiry (they previously had expires=None and never cleared/pruned). SPC radar cache files are pruned by age (RADAR_CACHE_RETENTION_DAYS, default 7). A new daily retention_cleanup_job prunes cleared sent_alerts (90d), AI records/attempts (30d), SPC pages past retained_until+30d, nws_alert_snapshots (180d), and disabled/revoked public tokens (30d) — each table in its own session/try-except, children before parents. Defaults are deliberately long and config-backed, aligned with the v2.0.0 Historical Explorer (#22) which will consume this history; windows documented in README.

Done in #97 (merged). MCD `SentAlert` rows now get a real 6h expiry (they previously had `expires=None` and never cleared/pruned). SPC radar cache files are pruned by age (`RADAR_CACHE_RETENTION_DAYS`, default 7). A new daily `retention_cleanup_job` prunes cleared `sent_alerts` (90d), AI records/attempts (30d), SPC pages past `retained_until`+30d, `nws_alert_snapshots` (180d), and disabled/revoked public tokens (30d) — each table in its own session/try-except, children before parents. Defaults are deliberately **long** and config-backed, aligned with the v2.0.0 Historical Explorer (#22) which will consume this history; windows documented in README.

claude-bot closed this issue

2026-07-18 18:18:00 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/WeatherBot#52

No description provided.

Rows
Columns