Parse NOAA's compact SPC timestamps, and take the fire issue time from the file date (#201) #202

Merged

claude-bot merged 1 commit from fix/spc-issue-time into main

2026-10-13 16:09:37 +00:00

claude-bot commented

2026-10-13 15:56:39 +00:00

Collaborator

Fixes #201, using option 1 as decided by @rbrooks on 2026-10-13. Option 2 (use NOAA's expire field instead of deriving expiry) is left as a possible follow-up.

The bug

NOAA's MapServer sends SPC issue times as twelve digits (202610131252). _parse_datetime only tried datetime.fromisoformat, which rejects that shape, so issued_at has been NULL on every outlook since SPC support was written: 0 of 75 rows on production, 0 of 194 on dev. The dashboard shows the request time as the "issued" time, public and impact pages show it blank, and each host logs about 1,900 warnings a day.

The change

_parse_datetime reads the compact YYYYMMDDHHMM form as UTC. An impossible date returns None rather than raising.
Fire outlooks take their issue time from idp_filedate instead of valid.

The second part is why this is not a one-line fix. Expiry is derived as issue date at 0800Z plus N days, and has only been roughly right because a NULL issue time fell back to "now".

Family	Issue-time source	Expiry after this PR
Convective (Days 1–3)	`issue` (real field)	Same as today, without the drift across 00Z
Extended (Days 4–8)	`issue`, else `valid` (unchanged)	Same as today
Fire (Days 1–2)	`idp_filedate` (was `valid`)	Same as today

The fire layers have no issue field. valid is when the outlook period starts, which for Day 2 is tomorrow, so had it simply started parsing, a Day 2 Critical/Extreme record would have been held a day too long. idp_filedate sits within minutes of issuance: 05:45:21Z for both fire products on 2026-10-13, against a Day 2 convective issue of 05:15Z and file date of 05:30Z.

If idp_filedate is ever missing, fire falls back to no issue time, which is today's behaviour.

What does not change

product_id for all three families. Dedup identity is untouched, so no existing record is re-sent.
Notification delivery.

What does change, visibly

"Issued" times on the dashboard, public pages, SPC impact pages and in AI summary context become the real SPC issue time.
An outlook polled after 00Z no longer has its expiry pushed out a day by the "now" fallback. Records for an outlook issued the previous UTC day now clear at 0800Z as the docstring always described.
The warning flood stops.

Existing rows are not backfilled. They age out, and product_id embeds the raw issue string if a backfill is ever wanted.

Verification

ruff: clean
Bulk tier: 1006 passed (1002 + 4 new), run locally on Python 3.14
Postgres tier: not run locally; left to this PR's CI. The change touches no database code.

New tests use property shapes copied from live MapServer responses, and cover: the compact format; a convective feature getting its issue time; a Day 2 fire outlook expiring on issue date + 2 rather than valid + 2; and the no-file-date fallback.

Correction to the record: my comment on #201 gave the fire file time as 07:45Z. It is 05:45Z. The reasoning there is otherwise unchanged.

🤖 Generated with Claude Code

Fixes #201, using option 1 as decided by @rbrooks on 2026-10-13. Option 2 (use NOAA's `expire` field instead of deriving expiry) is left as a possible follow-up. ## The bug NOAA's MapServer sends SPC issue times as twelve digits (`202610131252`). `_parse_datetime` only tried `datetime.fromisoformat`, which rejects that shape, so `issued_at` has been NULL on every outlook since SPC support was written: 0 of 75 rows on production, 0 of 194 on dev. The dashboard shows the request time as the "issued" time, public and impact pages show it blank, and each host logs about 1,900 warnings a day. ## The change 1. `_parse_datetime` reads the compact `YYYYMMDDHHMM` form as UTC. An impossible date returns `None` rather than raising. 2. Fire outlooks take their issue time from `idp_filedate` instead of `valid`. The second part is why this is not a one-line fix. Expiry is derived as **issue date at 0800Z plus N days**, and has only been roughly right because a NULL issue time fell back to "now". | Family | Issue-time source | Expiry after this PR | |---|---|---| | Convective (Days 1–3) | `issue` (real field) | Same as today, without the drift across 00Z | | Extended (Days 4–8) | `issue`, else `valid` (unchanged) | Same as today | | Fire (Days 1–2) | `idp_filedate` (was `valid`) | Same as today | The fire layers have no `issue` field. `valid` is when the outlook period starts, which for Day 2 is tomorrow, so had it simply started parsing, a Day 2 Critical/Extreme record would have been held a day too long. `idp_filedate` sits within minutes of issuance: 05:45:21Z for both fire products on 2026-10-13, against a Day 2 convective `issue` of 05:15Z and file date of 05:30Z. If `idp_filedate` is ever missing, fire falls back to no issue time, which is today's behaviour. ## What does not change - `product_id` for all three families. Dedup identity is untouched, so no existing record is re-sent. - Notification delivery. ## What does change, visibly - "Issued" times on the dashboard, public pages, SPC impact pages and in AI summary context become the real SPC issue time. - An outlook polled after 00Z no longer has its expiry pushed out a day by the "now" fallback. Records for an outlook issued the previous UTC day now clear at 0800Z as the docstring always described. - The warning flood stops. Existing rows are not backfilled. They age out, and `product_id` embeds the raw issue string if a backfill is ever wanted. ## Verification - ruff: clean - Bulk tier: **1006 passed** (1002 + 4 new), run locally on Python 3.14 - Postgres tier: not run locally; left to this PR's CI. The change touches no database code. New tests use property shapes copied from live MapServer responses, and cover: the compact format; a convective feature getting its issue time; a Day 2 fire outlook expiring on issue date + 2 rather than `valid` + 2; and the no-file-date fallback. Correction to the record: my comment on #201 gave the fire file time as 07:45Z. It is 05:45Z. The reasoning there is otherwise unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit

2026-10-13 15:56:39 +00:00

Parse NOAA's compact SPC timestamps, and take the fire issue time from the file date (#201 )

CI / test (pull_request) Successful in 7m23s

Details

af7ee6e183

SPC outlook issue times have never parsed. NOAA's MapServer sends them as
twelve digits (202610131252) and _parse_datetime only tried fromisoformat,
which rejects that shape. issued_at is NULL on every spc_outlook_pages row
on both hosts (0 of 75 on production, 0 of 194 on dev), the dashboard shows
the request time as the "issued" time, and each host logs about 1,900
warnings a day.

Parsing the value is one branch. The complication is that expiry is derived
as issue date at 0800Z plus N days, and has only been roughly right because
a NULL issue time fell back to "now".

- Convective and extended outlooks have a real `issue` field. Parsing it
  gives the expiry they get today, without the drift across 00Z.
- Fire outlooks have no `issue` field, and the parser used `valid` in its
  place. `valid` is when the outlook period starts, which for Day 2 is
  tomorrow, so once it parsed a Day 2 Critical/Extreme record would have
  been held a day too long. Fire now takes its issue time from
  `idp_filedate`, the feed's file timestamp, which sits within minutes of
  issuance (05:45Z for the 2026-10-13 products).

product_id is unchanged for all three families, so dedup identity is
untouched and no existing record is re-sent.

Using NOAA's own `expire` field instead of deriving expiry is the more
correct fix and is left as a follow-up: it changes when existing records
clear.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude-bot referenced this pull request

2026-10-13 15:56:44 +00:00

SPC outlook issue time has never parsed: issued_at is NULL on every row, and "issued" times shown are the fetch time #201

claude-bot merged commit 3666c23fa9 into main

2026-10-13 16:09:36 +00:00

claude-bot referenced this pull request from a commit

2026-10-13 16:09:38 +00:00

Merge pull request 'Parse NOAA's compact SPC timestamps, and take the fire issue time from the file date (#201)' (#202) from fix/spc-issue-time into main

claude-bot referenced this pull request

2026-10-13 16:10:33 +00:00

SPC outlook issue time has never parsed: issued_at is NULL on every row, and "issued" times shown are the fetch time #201

claude-bot referenced this pull request