Reminder polls crash on every run once a session has been warned (MissingGreenlet after rollback) #1115

New issue

Closed

opened 2026-10-10 17:55:51 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-10-10 17:55:51 +00:00

Collaborator

Found on 2026-10-10 reviewing dev's logs before the 4.5.0 release. Also on prod (4.4.1).

What

poll_session_at_risk has failed on dev on every run since 2026-10-06 17:56 UTC: 1,152 failures in four days. poll_session_reminders failed 23 times on dev in the same period, and 15 times on prod on 7 October (01:00 to 01:14 UTC, one run a minute, for a 7-day reminder).

asyncpg.exceptions.UniqueViolationError: duplicate key value violates unique constraint "uq_at_risk_session_confirmed_time"
...
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here.

Cause

Both tasks claim a "(session, time) already told" marker by inserting it and catching IntegrityError. The handler calls await db.rollback(), which expires every object the session holds, then reads session.id for the log line and carries on round the loop with expired session and campaign objects. Reading an expired attribute outside a greenlet raises MissingGreenlet, and the whole task dies (reminder_tasks.py around lines 2118, 2229, 2326, 2380, 2742).

So the already-warned path, which is the normal path on every poll after the first, crashes the poll. Every session later in that poll's list is not looked at while it lasts: for a reminder that is the length of the claim window; for an at-risk session it is until the session is played or stops being at risk.

Effect

At-risk warnings and reminders for other sessions can be late or missed while one already-warned session is in the list.
A full rollback also discards anything the same poll had done and not committed.
Log noise: about 290 tracebacks a day on dev.

Intended approach

Claim the marker inside a savepoint (async with db.begin_nested()), so a duplicate undoes only the insert and nothing is expired; read the ids into plain values before the claim.
The same for every except IntegrityError: await db.rollback() in a loop in app/tasks/.
Tests that fail today: a poll where the first session is already warned and a second is newly at risk warns the second and does not raise; the same shape for reminders. Check why the existing tests missed it (the tests' session probably does not expire on rollback the way task_session() does).

Found on 2026-10-10 reviewing dev's logs before the 4.5.0 release. **Also on prod (4.4.1).** ## What `poll_session_at_risk` has failed on dev on every run since 2026-10-06 17:56 UTC: 1,152 failures in four days. `poll_session_reminders` failed 23 times on dev in the same period, and 15 times on prod on 7 October (01:00 to 01:14 UTC, one run a minute, for a 7-day reminder). ``` asyncpg.exceptions.UniqueViolationError: duplicate key value violates unique constraint "uq_at_risk_session_confirmed_time" ... sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. ``` ## Cause Both tasks claim a "(session, time) already told" marker by inserting it and catching `IntegrityError`. The handler calls `await db.rollback()`, which expires every object the session holds, then reads `session.id` for the log line and carries on round the loop with expired `session` and `campaign` objects. Reading an expired attribute outside a greenlet raises `MissingGreenlet`, and the whole task dies (`reminder_tasks.py` around lines 2118, 2229, 2326, 2380, 2742). So the already-warned path, which is the normal path on every poll after the first, crashes the poll. Every session later in that poll's list is not looked at while it lasts: for a reminder that is the length of the claim window; for an at-risk session it is until the session is played or stops being at risk. ## Effect - At-risk warnings and reminders for other sessions can be late or missed while one already-warned session is in the list. - A full rollback also discards anything the same poll had done and not committed. - Log noise: about 290 tracebacks a day on dev. ## Intended approach - Claim the marker inside a savepoint (`async with db.begin_nested()`), so a duplicate undoes only the insert and nothing is expired; read the ids into plain values before the claim. - The same for every `except IntegrityError: await db.rollback()` in a loop in `app/tasks/`. - Tests that fail today: a poll where the first session is already warned and a second is newly at risk warns the second and does not raise; the same shape for reminders. Check why the existing tests missed it (the tests' session probably does not expire on rollback the way `task_session()` does).

rbrooks referenced this issue from a commit

2026-10-10 18:07:54 +00:00

fix(reminders): an already-sent marker no longer ends the poll (#1115)

claude-bot referenced this issue from a pull request that will close it,

2026-10-10 18:08:44 +00:00

fix(backend): the reminder and at-risk polls pass over what was already sent (#1115) #1118

claude-bot closed this issue

2026-10-10 18:44:31 +00:00

claude-bot referenced this issue from a commit

2026-10-10 18:44:31 +00:00

Merge PR #1118: the reminder and at-risk polls pass over what was already sent (#1115)

claude-bot commented

2026-10-10 18:44:37 +00:00

Author

Collaborator

Fixed in PR #1118, merged with CI green (14 checks). Claims run in a savepoint; both polls read each session by id as its turn comes; four new tests that commit real rows fail on the old code with MissingGreenlet and pass now. Goes to dev with v4.5.0-rc10, where the worker log is the check: poll_session_at_risk should stop failing. Not traced: import_jobs.Heartbeat after a failed heartbeat write.

claude-bot referenced this issue

2026-10-10 18:44:49 +00:00

chore(release): v4.5.0-rc10 #1119