rbrooks/WeatherBot

Watch

Fork 0

Real-world Qwen validation and deterministic validators #16

New issue

Closed

opened 2026-06-23 23:15:23 +00:00 by rbrooks · 2 comments

rbrooks commented

2026-06-23 23:15:23 +00:00

Owner

Target release: v1.3.0

AI summarization phases 1–3 are implemented; real-world model quality review is the outstanding longer-term item.

Acceptance

Track real-world Qwen output quality, uncertainty handling, validation-retry behavior, and latency over multiple real SPC/NWS events.
Turn repeated model mistakes found in review into explicit deterministic validators.
Continue tuning cloud-safe metadata defaults as real provider usage expands.

Source: docs/ROADMAP.md → In Progress → Local LLM Alert Summarization → "Longer-term review items".

**Target release:** v1.3.0 AI summarization phases 1–3 are implemented; real-world model quality review is the outstanding longer-term item. **Acceptance** - [ ] Track real-world Qwen output quality, uncertainty handling, validation-retry behavior, and latency over multiple real SPC/NWS events. - [ ] Turn repeated model mistakes found in review into explicit deterministic validators. - [ ] Continue tuning cloud-safe metadata defaults as real provider usage expands. Source: `docs/ROADMAP.md` → In Progress → Local LLM Alert Summarization → "Longer-term review items".

rbrooks added the

area:ai

type:feature

v1.3.0

labels

2026-06-23 23:16:46 +00:00

rbrooks added this to the v1.3.0 - AI summarization expansion milestone

2026-06-23 23:23:21 +00:00

claude-bot commented

2026-07-18 05:32:17 +00:00

Contributor

Audit note (2026-07-18)

From the 2026-07 codebase audit (docs/.internal/report-2026-07-17.md), finding F-33.

Add structural prompt hygiene to this issue's deterministic-validator scope:

Delimiter escaping: payload.source_text and source_title/source_metadata are interpolated into the <official_source> prompt delimiters unescaped (app/services/ai_summarizer.py:370-399). Escape/strip literal delimiter tokens from source text/title before interpolation so upstream (or malformed-feed) content can't break out of the source block.
Output URL validation: the validators check meteorology (app/services/ai_summarizer.py:592-770) but not links; add a validator that rejects/strips URLs in the output that don't appear in the source text.

Also within this issue's remit while touching the summarizer: _output_tokens_for_chars (app/services/ai_summarizer.py:189-200) is a no-op — max(4096, min(max_chars*6, 4096)) always returns 4096, so max_chars has no effect. Simplify to a documented constant or fix the intended floor.

## Audit note (2026-07-18) From the 2026-07 codebase audit (`docs/.internal/report-2026-07-17.md`), finding **F-33**. Add structural prompt hygiene to this issue's deterministic-validator scope: - **Delimiter escaping:** `payload.source_text` and `source_title`/`source_metadata` are interpolated into the `<official_source>` prompt delimiters unescaped (`app/services/ai_summarizer.py:370-399`). Escape/strip literal delimiter tokens from source text/title before interpolation so upstream (or malformed-feed) content can't break out of the source block. - **Output URL validation:** the validators check meteorology (`app/services/ai_summarizer.py:592-770`) but not links; add a validator that rejects/strips URLs in the output that don't appear in the source text. Also within this issue's remit while touching the summarizer: `_output_tokens_for_chars` (`app/services/ai_summarizer.py:189-200`) is a no-op — `max(4096, min(max_chars*6, 4096))` always returns 4096, so `max_chars` has no effect. Simplify to a documented constant or fix the intended floor.

claude-bot referenced this issue

2026-07-19 00:43:30 +00:00

Guardrailed timing/impact AI follow-ups (post-validation) #21

rbrooks referenced this issue from a commit

2026-07-19 00:59:57 +00:00

Summarizer: prompt hygiene, output-URL validator, quiet-hours REPLACE, guarded follow-ups

claude-bot referenced this issue from a pull request that will close it,

2026-07-19 01:00:11 +00:00

v1.3.0 Phase B: summarizer hygiene, quiet-hours mode, guarded follow-ups #105

claude-bot closed this issue

2026-07-19 01:07:18 +00:00

claude-bot commented

2026-07-19 01:07:28 +00:00

Contributor

Code-side work done in #105 (merged):

F-33 delimiter escaping — feed-controlled source_text/source_title are neutralized (zero-width space after < in delimiter tags) before interpolation into the prompt blocks; stored DB text is untouched.
F-33 output-URL validator — summary_validation_errors now rejects a URL in the summary that isn't present in the source text, feeding the existing retry loop.
F-33 token formula — _output_tokens_for_chars was a no-op (max(4096, min(max_chars*6, 4096)) ≡ 4096); fixed to max(4096, min(max_chars*6, 8192)).

Closing this issue: the deterministic-validator/hygiene deliverables are shipped. The remaining acceptance items ("track real-world Qwen quality over multiple real events", "turn repeated model mistakes into validators", "tune cloud-safe metadata defaults") are an ongoing operational activity with no fixed code deliverable — they depend on observing real SPC/NWS events against a running model. As specific, reproducible model mistakes surface in operation, file them as targeted validator issues (they'll be quick, isolated additions to summary_validation_errors). Tracking that indefinitely under one open issue isn't useful, so this closes with the milestone; reopen or file follow-ups as real findings appear.

Code-side work done in #105 (merged): - **F-33 delimiter escaping** — feed-controlled `source_text`/`source_title` are neutralized (zero-width space after `<` in delimiter tags) before interpolation into the prompt blocks; stored DB text is untouched. - **F-33 output-URL validator** — `summary_validation_errors` now rejects a URL in the summary that isn't present in the source text, feeding the existing retry loop. - **F-33 token formula** — `_output_tokens_for_chars` was a no-op (`max(4096, min(max_chars*6, 4096))` ≡ 4096); fixed to `max(4096, min(max_chars*6, 8192))`. Closing this issue: the deterministic-validator/hygiene deliverables are shipped. The remaining acceptance items ("track real-world Qwen quality over multiple real events", "turn repeated model mistakes into validators", "tune cloud-safe metadata defaults") are an **ongoing operational activity** with no fixed code deliverable — they depend on observing real SPC/NWS events against a running model. As specific, reproducible model mistakes surface in operation, file them as targeted validator issues (they'll be quick, isolated additions to `summary_validation_errors`). Tracking that indefinitely under one open issue isn't useful, so this closes with the milestone; reopen or file follow-ups as real findings appear.

claude-bot referenced this issue

2026-07-19 01:07:36 +00:00

Guardrailed timing/impact AI follow-ups (post-validation) #21

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/WeatherBot#16

No description provided.

Rows
Columns