A campaign's export carries handout files and maps (export schema 18) (#991) #1076

Merged

claude-bot merged 9 commits from feat/991-files-schema-18 into main 2026-10-04 10:32:52 +00:00

Conversation 0 Commits 9 Files changed 35 +5065 -93

claude-bot commented 2026-10-04 09:26:40 +00:00

Collaborator

Copy link

Closes #991. The last part of D273 (the owner's answer to Q145).

Summary

• Attachments travel with their page. A binned page's files travel only in wiki_bin.
• Maps. A new atlas section carries atlases, maps, markers, links and the trail, with each map's original image. Tiles are cut again on the new server.
• Audiences narrow. table and party are kept; anything else becomes GM-only, and a marker narrowed to GM loses its "revealed". A missing audience in the atlas section is GM-only.
• References resolve only among rows this import made.
• A failed import removes the new campaign's media directory.
• Schema gating. A file claiming 17 or less ignores the new sections and files; older files import as before.
• No migration, no bot change.

What is left out and listed

The import's answer gains files_left_out: [{kind, name, reason, size_bytes}], shown on the result screen. The export's README lists what it left out.

Reason	When
export / limit	the file would take the export or the import past the server's import limit
too_large	the file is over its per-file upload cap on this server
pixels	a map picture over 8192×6144 or with more than one frame; an attachment or page picture over 4000×3000
count	beyond 200 map pictures or 5,000 attachments in one import

A lying, corrupt, encrypted or wrongly-compressed entry still refuses the whole import. So does a file that is not an allowed type, and an entry named by more than one file.

Calls the plan did not cover

1. Map pictures are checked in the worker, after the import answers. The backend cannot hold the check: a 70 MB map peaks at about 400 MB alone, in a 512 MB container that is 340 to 425 MB warm. A picture that fails the full decode leaves its map without one, and the page says why.
2. The 8192×6144 bound. A 450 KB picture of 12000×12000 killed the process at 1 GB.
3. Over the per-file cap is left out, not refused, in a schema 18 file. A re-encoded file can be larger than its upload was, so a refusal could reject a server's own export.
4. The backend pixel check applies to files of every schema version. An older file with a page picture over 4000×3000 now imports without that picture, listed.
5. A map stuck "cutting" with no picture is failed after 8 hours, the existing lost-task threshold. Thirty minutes would fail maps still waiting their turn in a 200-map import.

Independent review

A read-only review of 605872e8 found no audience widening, no cross-campaign reference, no archive name reaching the filesystem and no stored-XSS route. It made eight findings; all are fixed in 25f13fbf, each with a test. Run against the earlier commit, the new tests gave 23 failed, 2 passed (the two are controls).

Changelog

• changelog.d/991-files-schema-18.added.md

Test plan

• Full backend suite at 605872e8, before the review fixes: 7272 passed, 14 skipped, 107 deselected.
• At the head commit: tests/export, both reader audits, the maps, tiles, files, handouts and pictures tests: 647 passed. The stuck-task watchdog: 10 passed. Lint clean. Version and changelog checks OK. The four touched frontend files: 63 passed, eslint 0 errors. All 16 coverage scripts exit 0.
• End to end through nginx, own compose project:
  • a map and an attachment made through the real upload routes, exported, imported and re-exported: media the same;
  • 205 small maps: 201 in 1.9 s, five listed as count, all 200 ready after 108 s;
  • an entry named twice: 422 in words;
  • a 17 KB PNG of 12000×12000 as a page picture and as an attachment: 201, both listed as pixels;
  • no restarts and no OOM kills in the backend or the worker.
• Memory, end to end: five 70 MB maps: backend peak 374 MB, worker peak 562 MB of 1 GB. Three 8192×6144 RGBA maps: worker peak 706 MB. These runs were before the review fixes; the decode path is unchanged.
• Not run: the full backend suite at the head commit and the full frontend suite (CI); synthetic and journeys.

Known and left for follow-ups

• Map jobs share the general worker. 200 maps occupied it for 108 s when small; large ones take far longer, and recordings queue behind them. A separate worker needs a seventh image tag and more memory on the host. To be filed.
• Page pictures have no count cap of their own. They are bounded by the 20,000-entry cap and the import budget.
• #1074 (the upload routes decode with no pixel bound) and #1075 (orphaned media directories).

🤖 Generated with Claude Code

Closes #991. The last part of D273 (the owner's answer to Q145). ## Summary - **Attachments** travel with their page. A binned page's files travel only in `wiki_bin`. - **Maps.** A new `atlas` section carries atlases, maps, markers, links and the trail, with each map's original image. Tiles are cut again on the new server. - **Audiences narrow.** `table` and `party` are kept; anything else becomes GM-only, and a marker narrowed to GM loses its "revealed". A missing audience in the atlas section is GM-only. - **References** resolve only among rows this import made. - **A failed import** removes the new campaign's media directory. - **Schema gating.** A file claiming 17 or less ignores the new sections and files; older files import as before. - No migration, no bot change. ## What is left out and listed The import's answer gains `files_left_out: [{kind, name, reason, size_bytes}]`, shown on the result screen. The export's README lists what it left out. | Reason | When | |---|---| | `export` / `limit` | the file would take the export or the import past the server's import limit | | `too_large` | the file is over its per-file upload cap on this server | | `pixels` | a map picture over 8192×6144 or with more than one frame; an attachment or page picture over 4000×3000 | | `count` | beyond 200 map pictures or 5,000 attachments in one import | A lying, corrupt, encrypted or wrongly-compressed entry still refuses the whole import. So does a file that is not an allowed type, and an entry named by more than one file. ## Calls the plan did not cover 1. **Map pictures are checked in the worker, after the import answers.** The backend cannot hold the check: a 70 MB map peaks at about 400 MB alone, in a 512 MB container that is 340 to 425 MB warm. A picture that fails the full decode leaves its map without one, and the page says why. 2. **The 8192×6144 bound.** A 450 KB picture of 12000×12000 killed the process at 1 GB. 3. **Over the per-file cap is left out**, not refused, in a schema 18 file. A re-encoded file can be larger than its upload was, so a refusal could reject a server's own export. 4. **The backend pixel check applies to files of every schema version.** An older file with a page picture over 4000×3000 now imports without that picture, listed. 5. **A map stuck "cutting" with no picture is failed after 8 hours**, the existing lost-task threshold. Thirty minutes would fail maps still waiting their turn in a 200-map import. ## Independent review A read-only review of `605872e8` found no audience widening, no cross-campaign reference, no archive name reaching the filesystem and no stored-XSS route. It made eight findings; all are fixed in `25f13fbf`, each with a test. Run against the earlier commit, the new tests gave `23 failed, 2 passed` (the two are controls). ## Changelog - [x] `changelog.d/991-files-schema-18.added.md` ## Test plan - [x] **Full backend suite** at `605872e8`, before the review fixes: `7272 passed, 14 skipped, 107 deselected`. - [x] **At the head commit:** `tests/export`, both reader audits, the maps, tiles, files, handouts and pictures tests: `647 passed`. The stuck-task watchdog: `10 passed`. Lint clean. Version and changelog checks OK. The four touched frontend files: `63 passed`, eslint 0 errors. All 16 coverage scripts exit 0. - [x] **End to end through nginx**, own compose project: - a map and an attachment made through the real upload routes, exported, imported and re-exported: media the same; - 205 small maps: 201 in 1.9 s, five listed as `count`, all 200 ready after 108 s; - an entry named twice: 422 in words; - a 17 KB PNG of 12000×12000 as a page picture and as an attachment: 201, both listed as `pixels`; - no restarts and no OOM kills in the backend or the worker. - [x] **Memory, end to end:** five 70 MB maps: backend peak 374 MB, worker peak 562 MB of 1 GB. Three 8192×6144 RGBA maps: worker peak 706 MB. These runs were before the review fixes; the decode path is unchanged. - [ ] **Not run:** the full backend suite at the head commit and the full frontend suite (CI); synthetic and journeys. ## Known and left for follow-ups - **Map jobs share the general worker.** 200 maps occupied it for 108 s when small; large ones take far longer, and recordings queue behind them. A separate worker needs a seventh image tag and more memory on the host. To be filed. - **Page pictures have no count cap of their own.** They are bounded by the 20,000-entry cap and the import budget. - #1074 (the upload routes decode with no pixel bound) and #1075 (orphaned media directories). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 8 commits 2026-10-04 09:26:41 +00:00

A campaign's export and import carry handout files and the map (export schema 18) (#991 part 2) ... 79c5d233cd

An export carried a page's picture and no other file: a handout came back
without its PDF, and no part of the map travelled.

- Each page's attachments are on the page's own object and travel with it;
  a binned page's files are only in the bin (media/bin/files/). On import
  each is read within the upload's cap, put through the upload's own check
  (files_service.save_attachment) and stored under a new random name in the
  new campaign's private store. An archive entry's name is only looked up,
  never used as a path.
- A new `atlas` section carries atlases, maps, markers, links and the
  parties' trails, with each map's original image. Tiles do not travel.
  References are remapped among the rows the import makes, or dropped.
  Audiences narrow: table and party are kept, everything else is the GMs',
  and a marker narrowed to the GMs is not revealed.
- A map's picture is not checked in the backend. Measured on the upload's
  check in a 512 MB container: a 10 MB JPEG peaked at 173 MB, a 70 MB PNG at
  403 MB, a 54 MB JPEG at 500 MB, and the backend runs at 340 to 425 MB. The
  import copies the file in pieces and the worker checks it after the commit
  (restore_imported_map), then cuts its tiles. A check that kills its worker
  is not tried again. A picture over 50 million pixels is left out and
  listed: a 450 KB PNG of 144 MP was killed at 512 MB and at 1 GB.
- The limit: an export is made to fit its server's import limit, and a file
  with no room is left out and listed in the README; an import leaves out a
  file that would take it past what the archive may unpack to. The import's
  answer lists both (files_left_out). A lying, damaged or invalid entry
  still refuses the import, and so does one over its upload's cap.
- A failed import removes the new campaign's media directory, on every
  path: a refusal on any entry, an error, a failed commit, a cancelled
  request. The removal is queued on the archive thread, behind any read
  still running there.
- A file that says it is older than 18 has its attachments and its atlas
  left unread. No migration, no bot change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The import's result screen lists the files that did not come, and a map says when its imported picture could not be used (#991 part 2) ... 94a324ada8

- ImportedBanner: under "Imported ...", each file left out with its page or
  map, what it was, its size and why (IS-488).
- The map page: a map waiting for its imported picture shows it as soon as
  the server has checked it, and a picture that did not pass is said to be
  unusable, not "still showing, just more slowly" (IS-489).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Docs, changelog and catalogue for export schema 18 (#991 part 2) ... 245b730849

- docs/API.md: the archive's media folders, `attachments` and `atlas`, what
  an import does with each, files_left_out and its three reasons, the new
  refusals and the 507 for a full media volume.
- docs/OPERATIONS.md: files in an archive and the media volume; why a map's
  picture is checked in the worker, with the measured figures.
- Ledger rows IS-488 and IS-489, classified in lanes 1b, 7 and 6d; the
  catalogue rebuilt with build.ps1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The worker takes one map's picture at a time (#991 part 2) ... d637e1fa97

An import queues a task per map and the worker runs two at once. Five 70 MB
maps imported together took the worker to 806 MB of its 1 GB, and two 49 MP
pictures decoded together would not have fitted. The check of an imported
picture and the cutting of tiles now hold a lock on the media volume, which
every worker process shares and the kernel releases if the process dies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

An imported map may be as large as an 8K export, and the worker hands back what each map took (#991 part 2) ... 773f8c2bb0

- The pixel bound is 8192x6144 (about 50 million), not a round 50 million
  that an 8K map export misses by half a percent. Measured: the check of a
  60 MB PNG of that size with transparency peaks at 515 MB in a bare
  interpreter, as a 49 MP JPEG does.
- After each map the worker process gives freed memory back to the system
  (malloc_trim). Without it each process that had decoded a map held on to
  what it freed: the idle worker stood at 425 MB after five maps, from 207.
  With the lock and this, five 70 MB maps peak at 562 MB and four 49 MP
  maps at 702 MB of the worker's 1 GB.
- docs/OPERATIONS.md has the figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The round trip is read as a player, too (#991 part 2) ... 605872e800

A player who joins the imported campaign reads the table's handout through
the private files route and neither of the narrowed pages' files, sees only
the table's atlas and markers, and gets a 404 for the GMs' map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge origin/main into feat/991-files-schema-18 6b8af1130f

An import of files is bounded, a file over its cap is left out, and a crash loses nothing (#991 part 2, review of schema 18) ... 25f13fbfdc

The owner's decision: a file over its per-file cap is left out and listed,
not a refusal, in a schema 18 file. And eight findings from the review.

1. How many. An import brings a picture for at most 200 maps and at most
   5,000 attachments; past that the rows come and the files are listed
   (reason `count`). An archive entry named as two files refuses the
   import. A map's picture has one frame, counted from its header, in the
   import and again in the worker.
2. A stored file can be over the cap its upload passed (it is the upload
   re-encoded). The export leaves it out by its own server's caps and says
   so in the README; the import leaves it out by its own (`too_large`).
3. A page's picture and an image attachment are decoded in the backend:
   their pixels over all frames are asked of the header first, and one over
   4000x3000 is left out and listed (`pixels`), in a file of any version.
   The upload routes are not changed (#1074).
4. The hourly sweep fails a map left "cutting" with no picture, and removes
   the file that was waiting. After the sweep's own 8 hours, not 30
   minutes: 200 maps wait their turn one at a time.
5. After a commit that did not return, the new campaign is looked up in a
   fresh session before its files are removed; if the look cannot be had,
   they stay.
6. The worker keeps a picture a GM uploaded while the check was queued,
   and an upload removes the import's waiting file.
7. A delivery that finds the checked file left behind by a crash, on a map
   that has its picture, cuts the tiles and does not report it lost.
8. A row of the map with no audience comes back as the GMs'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude-bot referenced this pull request 2026-10-04 09:26:56 +00:00

Map jobs share the general worker: a large import can hold up recordings #1077

rbrooks added 1 commit 2026-10-04 10:09:27 +00:00

The look after a lost commit uses task_session, not the pooled session (#991, #814) ... 6d4ec923d0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude-bot merged commit db1424e694 into main 2026-10-04 10:32:52 +00:00

claude-bot deleted branch feat/991-files-schema-18 2026-10-04 10:32:54 +00:00

claude-bot referenced this pull request from a commit 2026-10-04 10:32:54 +00:00

Merge PR #1076: a campaign's export carries handout files and maps (export schema 18) (#991)

claude-bot referenced this pull request 2026-10-04 10:33:01 +00:00

Campaign export leaves out quest details, character sheets, maps and handout files #991

claude-bot referenced this pull request 2026-10-04 10:33:50 +00:00

[Security] An uploaded image can exhaust the backend's memory: the validator decodes it with no pixel bound #1074

claude-bot referenced this pull request 2026-10-04 13:12:43 +00:00

A picture is checked by its pixels before it is opened, and a map upload is checked in the worker (#1074) #1078

claude-bot referenced this pull request 2026-10-04 16:37:57 +00:00

[Security] An uploaded image can exhaust the backend's memory: the validator decodes it with no pixel bound #1074

rbrooks referenced this pull request from a commit 2026-10-04 16:38:11 +00:00

chore(release): v4.5.0-rc4

claude-bot referenced this pull request 2026-10-04 16:38:16 +00:00

chore(release): v4.5.0-rc4 #1083

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backend

Issue related to the backend

  

bot

Issue related to the bot

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

frontend

Issue related to the frontend

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

recording

Issue related to the recording system

  

security

Security hardening / vulnerability work

  

wontfix

This won't be fixed

No labels

backend

bot

bug

duplicate

enhancement

frontend

help wanted

invalid

question

recording

security

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/Quest-Board!1076

No description provided.