A Redis blip answers signed-in requests with 500, not 503; and the OPERATIONS.md worker-healthcheck step doesn't work #1326

New issue

Closed

opened 2026-11-03 15:50:21 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-11-03 15:50:21 +00:00

Collaborator

Found by #1246's live checks (items 3 and 4) on a throwaway stack.

1. A Redis restart gives clients 500. While redis restarted under traffic, 18 of 2,269 signed-in requests got 500 {"detail":"Internal server error"} in about a third of a second. app/auth/session.py get_session (await r.get(key)) raised redis.exceptions.ConnectionError, uncaught, through get_current_user. Nobody was signed out, and the next requests worked.

Fix: catch Redis connection errors in the session read and answer 503 {"detail": "Quest Board is briefly unavailable. Try again in a moment."} (wording to the owner), so clients and the frontend treat it as "try again", not as a bug. It must never be a 401, which would sign people out. Test with a Redis client that raises.

2. The manual checklist's worker step is wrong (docs/OPERATIONS.md, "Verifying Healthchecks and AOF Persistence"):

docker compose exec worker kill -STOP 1 fails, because the image has no kill.
docker compose pause worker marks it unhealthy without running the probe (Docker 29), so it proves nothing.
Fix: use docker kill --signal=SIGSTOP <worker container>, expect unhealthy after about 80 s (three failed probes), and then --signal=SIGCONT.

A fix, not a feature, so it belongs to whichever release is next.

Found by #1246's live checks (items 3 and 4) on a throwaway stack. **1. A Redis restart gives clients 500.** While redis restarted under traffic, 18 of 2,269 signed-in requests got `500 {"detail":"Internal server error"}` in about a third of a second. `app/auth/session.py` `get_session` (`await r.get(key)`) raised `redis.exceptions.ConnectionError`, uncaught, through `get_current_user`. Nobody was signed out, and the next requests worked. - **Fix:** catch Redis connection errors in the session read and answer **503** `{"detail": "Quest Board is briefly unavailable. Try again in a moment."}` (wording to the owner), so clients and the frontend treat it as "try again", not as a bug. It must never be a 401, which would sign people out. Test with a Redis client that raises. **2. The manual checklist's worker step is wrong** (`docs/OPERATIONS.md`, "Verifying Healthchecks and AOF Persistence"): - `docker compose exec worker kill -STOP 1` fails, because the image has no `kill`. - `docker compose pause worker` marks it unhealthy without running the probe (Docker 29), so it proves nothing. - **Fix:** use `docker kill --signal=SIGSTOP <worker container>`, expect unhealthy after about 80 s (three failed probes), and then `--signal=SIGCONT`. A fix, not a feature, so it belongs to whichever release is next.