rbrooks/Quest-Board

Watch

Fork 0

Map jobs share the general worker: a large import can hold up recordings #1077

New issue

Closed

opened 2026-10-04 09:26:56 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-10-04 09:26:56 +00:00

Collaborator

Found by the independent review of #991 part 2 (PR #1076), and not fixed there.

What

Checking an imported map's picture and cutting a map's tiles run on the one Celery worker (--concurrency=2, default queue only), alongside process_audio, reminders and everything else. Map jobs take a lock so only one decodes at a time; the second pool slot then sits waiting on that lock for the next map job. While a batch of maps is queued, nothing else runs.

Measured: 200 small imported maps occupied the worker for 108 seconds.
Not measured: 200 large maps. Each one decodes up to 50 megapixels and writes up to 768 tiles, so the batch is likely over an hour.

#1076 caps an import at 200 map pictures, which bounds this. It does not remove it: a session's recording handed off during that time waits.

Options

One task per import that walks its maps in turn. The other pool slot stays free for everything else. No new service, no host change. Probably the right first step.
A second worker service for map jobs, on its own queue. Real isolation. It needs a seventh image tag in docker-compose.images.yml and check_version_sync.py, and about another 1 GB of memory on each host, so it is the owner's call.

Intended approach

Build option 1 unless the owner prefers option 2. Tests: with a batch of map jobs queued, a process_audio task starts without waiting for the batch.

#991, #1076, #1074.

Found by the independent review of #991 part 2 (PR #1076), and not fixed there. ## What Checking an imported map's picture and cutting a map's tiles run on the one Celery worker (`--concurrency=2`, default queue only), alongside `process_audio`, reminders and everything else. Map jobs take a lock so only one decodes at a time; the second pool slot then sits waiting on that lock for the next map job. While a batch of maps is queued, nothing else runs. - **Measured:** 200 small imported maps occupied the worker for 108 seconds. - **Not measured:** 200 large maps. Each one decodes up to 50 megapixels and writes up to 768 tiles, so the batch is likely over an hour. #1076 caps an import at 200 map pictures, which bounds this. It does not remove it: a session's recording handed off during that time waits. ## Options 1. **One task per import that walks its maps in turn.** The other pool slot stays free for everything else. No new service, no host change. Probably the right first step. 2. **A second worker service for map jobs, on its own queue.** Real isolation. It needs a seventh image tag in `docker-compose.images.yml` and `check_version_sync.py`, and about another 1 GB of memory on each host, so it is the owner's call. ## Intended approach Build option 1 unless the owner prefers option 2. Tests: with a batch of map jobs queued, a `process_audio` task starts without waiting for the batch. ## Related #991, #1076, #1074.

claude-bot referenced this issue

2026-10-04 10:33:01 +00:00

Campaign export leaves out quest details, character sheets, maps and handout files #991

claude-bot referenced this issue

2026-10-04 16:37:57 +00:00

[Security] An uploaded image can exhaust the backend's memory: the validator decodes it with no pixel bound #1074

claude-bot commented

2026-10-04 17:19:00 +00:00

Author

Collaborator

Picking this up, on fix/1077-map-jobs-one-per-import. Option 1: no new service and no compose change.

Planned

An import enqueues one task that walks its maps in turn, in place of one task per map.
A map task that finds the lock held gives its pool slot back and retries later. Today it waits in the slot.
The hourly sweep must not fail a map that is still waiting its turn in a live batch.

To be shown in a running stack: a quick non-map task started during a batch of about 30 maps starts within seconds, measured against main.

Option 2 (a separate worker) stays the owner's call.

**Picking this up**, on `fix/1077-map-jobs-one-per-import`. Option 1: no new service and no compose change. **Planned** - An import enqueues one task that walks its maps in turn, in place of one task per map. - A map task that finds the lock held gives its pool slot back and retries later. Today it waits in the slot. - The hourly sweep must not fail a map that is still waiting its turn in a live batch. **To be shown in a running stack:** a quick non-map task started during a batch of about 30 maps starts within seconds, measured against main. Option 2 (a separate worker) stays the owner's call.

rbrooks referenced this issue from a commit

2026-10-04 18:26:01 +00:00

An import's maps are one worker task, and no map task waits in a slot (#1077)

claude-bot referenced this issue from a pull request that will close it,

2026-10-04 18:58:46 +00:00

An import's maps are one job that takes turns, so a batch no longer holds up the worker (#1077) #1085

claude-bot referenced this issue from a commit

2026-10-04 19:25:25 +00:00

Merge PR #1085: an import's maps are one job that takes turns, so a batch no longer holds up the worker (#1077)

claude-bot closed this issue

2026-10-04 19:25:25 +00:00

claude-bot commented

2026-10-04 19:25:32 +00:00

Author

Collaborator

Fixed in PR #1085, merged to main. CI was green on the PR's head.

The change: an import enqueues one task that walks its maps in turn, and a map task that finds the lock held gives its pool slot back and retries.
Measured in a running stack (a real import of 30 maps of 4000×3000): a 10 ms task sent during the batch completed in 0.08 to 0.26 s, against 14 to 34 s on main. Batch time is unchanged within noise.
A kill mid-batch was picked up again in 0.2 s and all 30 maps finished.

Not shown in the stack: an upload during a batch and two imports at once (unit tests with real locks only); 200 large maps were not measured.

Still the owner's call: a separate worker for map jobs (option 2). Nothing here needs it.

Reaches dev with the next prerelease.

**Fixed in PR #1085, merged to main. CI was green on the PR's head.** - **The change:** an import enqueues one task that walks its maps in turn, and a map task that finds the lock held gives its pool slot back and retries. - **Measured in a running stack** (a real import of 30 maps of 4000×3000): a 10 ms task sent during the batch completed in 0.08 to 0.26 s, against 14 to 34 s on main. Batch time is unchanged within noise. - **A kill mid-batch** was picked up again in 0.2 s and all 30 maps finished. **Not shown in the stack:** an upload during a batch and two imports at once (unit tests with real locks only); 200 large maps were not measured. **Still the owner's call:** a separate worker for map jobs (option 2). Nothing here needs it. Reaches dev with the next prerelease.