An import's maps are one job that takes turns, so a batch no longer holds up the worker (#1077) #1085

Merged

claude-bot merged 1 commit from fix/1077-map-jobs-one-per-import into main 2026-10-04 19:25:25 +00:00

Conversation 0 Commits 1 Files changed 10 +1583 -103

claude-bot commented 2026-10-04 18:58:46 +00:00

Collaborator

Copy link

Closes #1077. Option 1 of the issue: no new service, queue, compose change, migration or bot change.

Summary

• One task per import. restore_imported_maps(campaign_id) walks the campaign's waiting maps in order, taking the map lock per map. Each map still flips to ready or failed as it is done.
• A map task that finds the lock held gives its slot back. It polls for at most 3 s, then retries later (5 s × 1.5ⁿ, capped at 30 s, with jitter). Before, it waited in the pool slot, so a batch took both slots.
• An upload gets a turn between two maps of a batch, through a small gate file taken before the lock.
• One walker per campaign. A duplicate delivery finds the first at work and returns.
• The hourly sweep knows a walk is alive by its state file's age. It leaves that campaign's maps alone, re-queues a walk that has been quiet for 15 minutes, and fails the maps only past the 8-hour cutoff.

Three things beyond the issue

1. The walker gives its slot back every 30 s and rejoins the queue. Otherwise a long transcription in the other slot would starve reminders for the length of a batch.
2. The sweep re-queues a quiet walk. A worker killed outright keeps its unacknowledged message for days; with one message per import that would lose the rest of the batch.
3. A map killed during tile cutting is tried once more, then failed.

Bounds

What	Bound	Then
A per-map task asking for the lock	360 retries, about 3 hours	the map is failed with a sentence and its pending file removed
A walker yielding with no map completed	360 in a row	the remaining maps are failed
A walker's deliveries dying with no map completed	6 in a row	the remaining maps are failed

Measured in a running stack

Own compose project on the dev host, worker at 1 GB with two slots. A real import of 30 maps of 4000×3000 (192 tiles each). The probe is a 10 ms read-only task sent during the batch.

	main	this branch
Probe sent at 5 s completed after	34.37 s	0.26 s
Probe sent at 15 s completed after	24.30 s	0.08 s
Probe sent at 25 s completed after	14.19 s	0.09 s
Batch time	40.84 s	35.72 s, 41.64 s, 45.48 s (three runs)
Worker peak memory	585 MB	539 MB, 534 MB, 579 MB
Map tasks received	30	1 task, 2 deliveries

Kill run: the pool process holding the lock was killed 6 s into the walk, during tile cutting. The task was received again 0.2 s later and all 30 maps ended ready in 41.64 s; the probes still completed in under 0.2 s.

Changelog

• changelog.d/1077-map-jobs-one-per-import.fixed.md

Test plan

• Full backend suite at the head commit: 7420 passed, 14 skipped, 107 deselected.
• New tests/export/test_map_jobs_one_per_import.py, 23 tests, with real file locks: one task per import; a held lock makes a task yield in under 3 s; kills during the check and during tile cutting; the sweep's three cases; both retry bounds.
• Lint clean; version and changelog checks OK.
• Not run: an upload during a batch, a kill during the picture check, and two imports at once were shown by unit tests only, not in the stack. 200 large maps were not measured. The new tests were not run against the old code; the table above is the before and after.
• Frontend, bot and synthetic suites: untouched; CI.

Known limit

On a media volume without flock, one import's maps are still taken in turn, but an upload or a second import can decode alongside. docs/OPERATIONS.md says so.

🤖 Generated with Claude Code

Closes #1077. Option 1 of the issue: no new service, queue, compose change, migration or bot change. ## Summary - **One task per import.** `restore_imported_maps(campaign_id)` walks the campaign's waiting maps in order, taking the map lock per map. Each map still flips to ready or failed as it is done. - **A map task that finds the lock held gives its slot back.** It polls for at most 3 s, then retries later (5 s × 1.5ⁿ, capped at 30 s, with jitter). Before, it waited in the pool slot, so a batch took both slots. - **An upload gets a turn between two maps of a batch**, through a small gate file taken before the lock. - **One walker per campaign.** A duplicate delivery finds the first at work and returns. - **The hourly sweep knows a walk is alive** by its state file's age. It leaves that campaign's maps alone, re-queues a walk that has been quiet for 15 minutes, and fails the maps only past the 8-hour cutoff. ## Three things beyond the issue 1. **The walker gives its slot back every 30 s** and rejoins the queue. Otherwise a long transcription in the other slot would starve reminders for the length of a batch. 2. **The sweep re-queues a quiet walk.** A worker killed outright keeps its unacknowledged message for days; with one message per import that would lose the rest of the batch. 3. **A map killed during tile cutting is tried once more, then failed.** ## Bounds | What | Bound | Then | |---|---|---| | A per-map task asking for the lock | 360 retries, about 3 hours | the map is failed with a sentence and its pending file removed | | A walker yielding with no map completed | 360 in a row | the remaining maps are failed | | A walker's deliveries dying with no map completed | 6 in a row | the remaining maps are failed | ## Measured in a running stack Own compose project on the dev host, worker at 1 GB with two slots. A real import of 30 maps of 4000×3000 (192 tiles each). The probe is a 10 ms read-only task sent during the batch. | | main | this branch | |---|---|---| | Probe sent at 5 s completed after | 34.37 s | 0.26 s | | Probe sent at 15 s completed after | 24.30 s | 0.08 s | | Probe sent at 25 s completed after | 14.19 s | 0.09 s | | Batch time | 40.84 s | 35.72 s, 41.64 s, 45.48 s (three runs) | | Worker peak memory | 585 MB | 539 MB, 534 MB, 579 MB | | Map tasks received | 30 | 1 task, 2 deliveries | **Kill run:** the pool process holding the lock was killed 6 s into the walk, during tile cutting. The task was received again 0.2 s later and all 30 maps ended ready in 41.64 s; the probes still completed in under 0.2 s. ## Changelog - [x] `changelog.d/1077-map-jobs-one-per-import.fixed.md` ## Test plan - [x] **Full backend suite** at the head commit: `7420 passed, 14 skipped, 107 deselected`. - [x] New `tests/export/test_map_jobs_one_per_import.py`, 23 tests, with real file locks: one task per import; a held lock makes a task yield in under 3 s; kills during the check and during tile cutting; the sweep's three cases; both retry bounds. - [x] Lint clean; version and changelog checks OK. - [ ] **Not run:** an upload during a batch, a kill during the picture check, and two imports at once were shown by unit tests only, not in the stack. 200 large maps were not measured. The new tests were not run against the old code; the table above is the before and after. - [ ] Frontend, bot and synthetic suites: untouched; CI. ## Known limit On a media volume without `flock`, one import's maps are still taken in turn, but an upload or a second import can decode alongside. `docs/OPERATIONS.md` says so. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-10-04 18:58:47 +00:00

An import's maps are one worker task, and no map task waits in a slot (#1077) ... 3f3594df2c

The worker is one Celery pool of two slots and runs everything. Maps are
taken one at a time, by a lock, because two large pictures do not fit in
its 1 GB. An import queued a task per map, so with a batch queued one slot
decoded a map while the other held the next map's task, waiting for the
lock: both slots were the maps', and a recording handed off meanwhile, or
a reminder, waited for the whole batch.

- An import queues one task, restore_imported_maps(campaign_id), which
  takes that import's maps in a stable order with the per-map code that
  was there, under the lock for each map and not across the walk. Each
  map goes to ready or failed as it is done, as before.
- one_map_at_a_time no longer waits for long. It asks for the lock for
  3 seconds and then raises; the task (an upload's restore_imported_map,
  cut_map_tiles, a second import's walk) asks Celery to deliver it again
  in 5 to 30 seconds, with jitter, and gives its slot back. A per-map
  task asks 360 times (about three hours), then fails the map in words.
- A gate file beside the lock, taken first and let go once the lock is
  held, so a waiter gets its turn between two maps of a walk.
- The walk goes to the back of the queue every 30 seconds, between two
  maps, so a reminder gets a turn when the other slot is on a long
  transcription.
- One walk per campaign, by a lock of its own that is never waited for.
- The walk's place is on disk (media/private/.map-imports/<id>.json),
  written at every step. A redelivery after a kill fails the picture
  whose check was begun without opening it again, tries a map that died
  while its tiles were cut once more, and goes on with the rest. Six
  deaths in a row with no map done between them fail what is left.
- The hourly sweep reads that file's age: under 8 hours the walk is
  alive and its maps are let be whatever their age; over 15 minutes it
  is queued again, in case its task was lost with a worker killed
  outright; over 8 hours the maps left are failed and the file removed.

Measured on a throwaway stack, 30 maps of 4000x3000, a task that is not a
map's sent 5, 15 and 25 s into the batch. Before: all three ran 39 s in,
after 29 maps; batch 41 s; worker peak 585 MB. After: each ran within
0.3 s; batch 36 to 45 s over three runs; peak 534 to 579 MB. No oom kill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude-bot merged commit d9b33ce045 into main 2026-10-04 19:25:25 +00:00

claude-bot referenced this pull request from a commit 2026-10-04 19:25:25 +00:00

Merge PR #1085: an import's maps are one job that takes turns, so a batch no longer holds up the worker (#1077)

claude-bot deleted branch fix/1077-map-jobs-one-per-import 2026-10-04 19:25:26 +00:00

claude-bot referenced this pull request 2026-10-04 19:25:32 +00:00

Map jobs share the general worker: a large import can hold up recordings #1077

rbrooks referenced this pull request from a commit 2026-10-05 00:39:29 +00:00

chore(release): v4.5.0-rc5

claude-bot referenced this pull request 2026-10-05 00:39:32 +00:00

chore(release): v4.5.0-rc5 #1092

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backend

Issue related to the backend

  

bot

Issue related to the bot

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

frontend

Issue related to the frontend

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

recording

Issue related to the recording system

  

security

Security hardening / vulnerability work

  

wontfix

This won't be fixed

No labels

backend

bot

bug

duplicate

enhancement

frontend

help wanted

invalid

question

recording

security

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/Quest-Board!1085

No description provided.