Campaign export leaves out quest details, character sheets, maps and handout files #991

Closed
opened 2026-10-01 17:04:41 +00:00 by claude-bot · 6 comments
Collaborator

Found while fixing #830, which added prep sheets, story arcs and the shelf. The same kind of gap remains for things lanes 8 and 11 built:

  • Quest details (quest_details, objectives, giver, reward, status). A quest page comes back as a plain page with no quest data at all.
  • Character sheets and stat blocks (character_sheets, their versions, kept rolls). Not in the file. A stat block's projection in LoreEntry.stats may travel with its page, but the sheet behind it does not, so the block cannot be edited or shown by field after an import. #830 carries stat-block reveals with their frozen party text; the sheet's own per-group "shown" state does not travel.
  • Maps (atlases, maps, tiles, markers, the trail).
  • Handout attachment files, and whether private media (media/private/…) travels at all.

A GM who exports to move servers or to keep a copy loses all of these today.

To do. For each: add it to the export and the import with references remapped and audiences never widened (follow prep_shelf_archive.py and discussion_archive.py), bump the schema version, keep older files importing, and extend the round-trip tests. Files (maps' images and tiles, handouts) need a decision on size limits for the zip.

Smaller leftovers from #830, to fold in or leave:

  • Workbench generation runs do not travel, so History is empty after an import and an unpinned generated prep item no longer recognises its own shelf card ("Add to shelf" would make a second card).
  • Rule cards come back reading from their snapshot, not re-linked to a pack on the new server.

Refs #830, #734, #737.

Found while fixing #830, which added prep sheets, story arcs and the shelf. The same kind of gap remains for things lanes 8 and 11 built: - **Quest details** (`quest_details`, objectives, giver, reward, status). A quest page comes back as a plain page with no quest data at all. - **Character sheets and stat blocks** (`character_sheets`, their versions, kept rolls). Not in the file. A stat block's projection in `LoreEntry.stats` may travel with its page, but the sheet behind it does not, so the block cannot be edited or shown by field after an import. #830 carries stat-block reveals with their frozen party text; the sheet's own per-group "shown" state does not travel. - **Maps** (atlases, maps, tiles, markers, the trail). - **Handout attachment files**, and whether private media (`media/private/…`) travels at all. A GM who exports to move servers or to keep a copy loses all of these today. **To do.** For each: add it to the export and the import with references remapped and audiences never widened (follow `prep_shelf_archive.py` and `discussion_archive.py`), bump the schema version, keep older files importing, and extend the round-trip tests. Files (maps' images and tiles, handouts) need a decision on size limits for the zip. **Smaller leftovers from #830**, to fold in or leave: - Workbench generation runs do not travel, so History is empty after an import and an unpinned generated prep item no longer recognises its own shelf card ("Add to shelf" would make a second card). - Rule cards come back reading from their snapshot, not re-linked to a pack on the new server. Refs #830, #734, #737.
Author
Collaborator

Part 1 (quest details, character sheets, parties; export schema 17) is in PR #1053. It has been brought up to main and privacy-reviewed, and CI is running.

Owner's answer to Q145 (X5), 2026-10-03: (a), as recommended. This is part 2:

  • Carry handout files and each map's original image. Tiles are not carried; they are cut again on the new server.
  • Write and read the zip on disk, not in memory.
  • Raise the import limit to 1 GB, settable in Admin.
  • A file that would go over the limit is left out and listed, with its page.

To be recorded as D273. Part 2 starts once #1053 is merged. It also covers #1054 (the import reads data.json with no cap on its decompressed size): reading the zip on disk through bounded readers is the same work.

**Part 1** (quest details, character sheets, parties; export schema 17) is in PR #1053. It has been brought up to main and privacy-reviewed, and CI is running. **Owner's answer to Q145 (X5), 2026-10-03: (a), as recommended.** This is part 2: - Carry handout files and each map's **original image**. Tiles are not carried; they are cut again on the new server. - Write and read the zip **on disk**, not in memory. - Raise the import limit to **1 GB, settable in Admin**. - A file that would go over the limit is **left out and listed, with its page**. To be recorded as D273. Part 2 starts once #1053 is merged. It also covers #1054 (the import reads `data.json` with no cap on its decompressed size): reading the zip on disk through bounded readers is the same work.
Author
Collaborator

Part 1 merged: PR #1053, 2026-10-03, CI green. Export schema 17 carries quest details, character sheets with their versions and kept rolls, and parties with membership over time. It reaches dev with the next prerelease.

The privacy review before the merge found one defect, fixed in the PR: an imported quest had no size limits. It also filed #1054 (the uncapped read of data.json), which part 2 closes.

Part 2 is being planned now, per the owner's Q145 answer above. It is two PRs: the zip on disk with bounded reads and the Admin import limit first, then the files (handout attachments and map originals, schema 18).

**Part 1 merged:** PR #1053, 2026-10-03, CI green. Export schema 17 carries quest details, character sheets with their versions and kept rolls, and parties with membership over time. It reaches dev with the next prerelease. The privacy review before the merge found one defect, fixed in the PR: an imported quest had no size limits. It also filed #1054 (the uncapped read of `data.json`), which part 2 closes. **Part 2 is being planned now**, per the owner's Q145 answer above. It is two PRs: the zip on disk with bounded reads and the Admin import limit first, then the files (handout attachments and map originals, schema 18).
Author
Collaborator

Part 2 plan (D273), as two PRs

Planned from origin/main at the #1053 merge. The planning turned up a live bug, filed and being fixed separately as #1057: nginx refuses every upload over 1 MB.

PR A: plumbing, no new content (closes #1054)

  • Import.
    • The upload is spooled to an anonymous temp file as it arrives. Today UploadFile spools the whole body with no cap, then 100 MB of it is read into memory.
    • Entries are read through bounded readers: declared size checked first, then real bytes counted in chunks. A lying header, an overlapping-entry bomb, an entry-count bomb, an encrypted entry or a duplicate name is refused with a 422 in words.
    • data.json has its own fixed cap.
  • Export. The zip is written to a temp file and streamed back with a length. Today the zip and every picture are built in memory.
  • The limit. An admin setting (default 1 GB) on Admin → Import a campaign, enforced by the backend as the upload is received (413 in words). nginx gives the import route its own location and leaves the limit to the backend.
  • Temp files. They are anonymous: unlinked at creation, so no failure path leaves one behind.
  • Migration. None.

PR B: schema 18, the files

  • Handout attachments. They sit on their page's object, so a binned page's files travel only in wiki_bin.
  • Maps. A new atlas section carries atlases, maps, markers, links and the trail. Maps are not exported at all today. Each map's original image travels; tiles are cut again after the import commits.
  • Audiences. They narrow as everywhere else: table and party are kept, everything else becomes GM-only, and a marker narrowed to GM loses its "revealed".
  • Over the limit. The file is left out and listed with its page, in the export's README and on the import's result screen. A lying or corrupt entry still refuses the whole import.
  • A failed import removes the new campaign's media directory. Today it is left there.
  • Schema gating. 18-only sections are ignored in a file claiming 17 or less.
  • Migration. None.

Choices made where the decision did not say

Flag any of these and they change.

  • "1 GB" applies to both the upload and the decompressed total. Media is stored uncompressed, so they are nearly the same number.
  • The export's budget is the exporting server's own setting.
  • An invalid file (not really an image or a PDF) refuses the import, as a bad page picture does today. Only over-limit files are "left out and listed".
  • The setting's range is 100 MB to 4 GB.
  • The export download becomes a plain link, not a blob held in the browser's memory.

Left out of part 2

Workbench generation runs and rule cards re-linking to a pack. Neither involves files. The first needs its own privacy review (prompts, provider data, usage); the second needs pack matching like sheet definitions. Each gets its own issue.

Risks

  • Memory. Re-encoding a 75 MB map in the 512 MB backend, many in a row. They run one at a time; the fallback is validating in the worker, which has 1 GB.
  • Time. A 1 GB import can outlast nginx's 300 s read timeout. The import location gets longer timeouts. A background import job is out of scope.
## Part 2 plan (D273), as two PRs Planned from `origin/main` at the #1053 merge. The planning turned up a live bug, filed and being fixed separately as #1057: nginx refuses every upload over 1 MB. ### PR A: plumbing, no new content (closes #1054) - **Import.** - The upload is spooled to an anonymous temp file as it arrives. Today `UploadFile` spools the whole body with no cap, then 100 MB of it is read into memory. - Entries are read through bounded readers: declared size checked first, then real bytes counted in chunks. A lying header, an overlapping-entry bomb, an entry-count bomb, an encrypted entry or a duplicate name is refused with a 422 in words. - `data.json` has its own fixed cap. - **Export.** The zip is written to a temp file and streamed back with a length. Today the zip and every picture are built in memory. - **The limit.** An admin setting (default 1 GB) on Admin → Import a campaign, enforced by the backend as the upload is received (413 in words). nginx gives the import route its own location and leaves the limit to the backend. - **Temp files.** They are anonymous: unlinked at creation, so no failure path leaves one behind. - **Migration.** None. ### PR B: schema 18, the files - **Handout attachments.** They sit on their page's object, so a binned page's files travel only in `wiki_bin`. - **Maps.** A new `atlas` section carries atlases, maps, markers, links and the trail. Maps are not exported at all today. Each map's original image travels; tiles are cut again after the import commits. - **Audiences.** They narrow as everywhere else: `table` and `party` are kept, everything else becomes GM-only, and a marker narrowed to GM loses its "revealed". - **Over the limit.** The file is left out and listed with its page, in the export's README and on the import's result screen. A lying or corrupt entry still refuses the whole import. - **A failed import** removes the new campaign's media directory. Today it is left there. - **Schema gating.** 18-only sections are ignored in a file claiming 17 or less. - **Migration.** None. ### Choices made where the decision did not say Flag any of these and they change. - **"1 GB" applies to both** the upload and the decompressed total. Media is stored uncompressed, so they are nearly the same number. - **The export's budget** is the exporting server's own setting. - **An invalid file** (not really an image or a PDF) refuses the import, as a bad page picture does today. Only over-limit files are "left out and listed". - **The setting's range** is 100 MB to 4 GB. - **The export download** becomes a plain link, not a blob held in the browser's memory. ### Left out of part 2 Workbench generation runs and rule cards re-linking to a pack. Neither involves files. The first needs its own privacy review (prompts, provider data, usage); the second needs pack matching like sheet definitions. Each gets its own issue. ### Risks - **Memory.** Re-encoding a 75 MB map in the 512 MB backend, many in a row. They run one at a time; the fallback is validating in the worker, which has 1 GB. - **Time.** A 1 GB import can outlast nginx's 300 s read timeout. The import location gets longer timeouts. A background import job is out of scope.
Author
Collaborator

Part 2, PR A is merged: PR #1072. CI was green on the merged commit.

  • Export and import are handled on disk. A 1,000 MB import went through nginx with the backend peaking at 349 MB.
  • The largest import is an Admin setting, 1 GB by default.
  • The import form shows the server's reason when it refuses.
  • An independent security review found five defects before the merge; all five are fixed and tested. Details are on the PR and on #1054, which stays open for parsing in the worker.

Remaining on this issue: PR B (export schema 18)

  • handout files;
  • map originals.

Split out

  • #1069: Workbench generation runs.
  • #1070: rule cards re-linking to their pack.
**Part 2, PR A is merged: PR #1072.** CI was green on the merged commit. - **Export and import are handled on disk.** A 1,000 MB import went through nginx with the backend peaking at 349 MB. - **The largest import is an Admin setting,** 1 GB by default. - **The import form shows the server's reason** when it refuses. - An independent security review found five defects before the merge; all five are fixed and tested. Details are on the PR and on #1054, which stays open for parsing in the worker. **Remaining on this issue: PR B** (export schema 18) - handout files; - map originals. **Split out** - #1069: Workbench generation runs. - #1070: rule cards re-linking to their pack.
Author
Collaborator

PR B (schema 18: handout files and maps) is built on feat/991-files-schema-18 and under review. Not yet a PR.

Built (head 605872e8; full backend suite 7272 passed)

  • Attachments travel with their page; a binned page's files travel only in wiki_bin.
  • A new atlas section: atlases, maps, markers, links and the trail, with each map's original image.
  • Audiences narrow; references are remapped to rows this import made.
  • A failed import removes the new campaign's media directory.

Three calls the plan did not cover

  1. Map pictures are checked in the worker, after the import answers. The backend cannot hold the check: a 70 MB map peaks at about 400 MB alone. A picture that fails the full decode leaves its map without one, and the map's page says why. Accepted; it was the plan's named fallback.
  2. A map picture over 8192×6144 is left out and listed. A 450 KB picture of 12000×12000 killed the process even at 1 GB. Accepted; the owner may change the number.
  3. A file over its per-file cap is left out and listed, not a refusal of the whole import. This follows the owner's decision for the overall limit (D273). A corrupt or lying entry still refuses.

Independent review of 605872e8: eight findings, being fixed

  • Sound: no audience widening, no cross-campaign reference, no path from the file reaching the filesystem, and no stored-XSS route.
  • Medium-high: no cap on maps per import. One file could queue about 45,000 map tasks and write tens of millions of tiles. Fix: 200 map pictures and 5,000 attachments per import, one zip entry per file, and frames counted in the pixel bound.
  • Medium: a genuine export could be refused by the server that made it, because a re-encoded file can end up over the upload cap. Fixed by item 3 above.
  • Low: six smaller ones.
    • Imported attachments and page pictures get a pixel check before decoding.
    • A map stuck "cutting" is failed after a bound.
    • A commit that landed but was not seen no longer discards a live campaign's media.
    • The worker no longer overwrites a picture uploaded meanwhile.
    • A redelivery no longer marks a map that has a picture as having none.
    • A missing audience in the atlas section becomes GM-only.

Filed from this work: #1074, an uploaded image can exhaust the backend's memory (already on main, in the ordinary upload routes).

Not built, to be filed: a sweep for media directories left behind when the backend dies mid-import.

**PR B (schema 18: handout files and maps) is built on `feat/991-files-schema-18` and under review. Not yet a PR.** **Built** (head `605872e8`; full backend suite `7272 passed`) - Attachments travel with their page; a binned page's files travel only in `wiki_bin`. - A new `atlas` section: atlases, maps, markers, links and the trail, with each map's original image. - Audiences narrow; references are remapped to rows this import made. - A failed import removes the new campaign's media directory. **Three calls the plan did not cover** 1. **Map pictures are checked in the worker, after the import answers.** The backend cannot hold the check: a 70 MB map peaks at about 400 MB alone. A picture that fails the full decode leaves its map without one, and the map's page says why. Accepted; it was the plan's named fallback. 2. **A map picture over 8192×6144 is left out and listed.** A 450 KB picture of 12000×12000 killed the process even at 1 GB. Accepted; the owner may change the number. 3. **A file over its per-file cap is left out and listed**, not a refusal of the whole import. This follows the owner's decision for the overall limit (D273). A corrupt or lying entry still refuses. **Independent review of `605872e8`: eight findings, being fixed** - **Sound:** no audience widening, no cross-campaign reference, no path from the file reaching the filesystem, and no stored-XSS route. - **Medium-high:** no cap on maps per import. One file could queue about 45,000 map tasks and write tens of millions of tiles. Fix: 200 map pictures and 5,000 attachments per import, one zip entry per file, and frames counted in the pixel bound. - **Medium:** a genuine export could be refused by the server that made it, because a re-encoded file can end up over the upload cap. Fixed by item 3 above. - **Low:** six smaller ones. - Imported attachments and page pictures get a pixel check before decoding. - A map stuck "cutting" is failed after a bound. - A commit that landed but was not seen no longer discards a live campaign's media. - The worker no longer overwrites a picture uploaded meanwhile. - A redelivery no longer marks a map that has a picture as having none. - A missing audience in the atlas section becomes GM-only. **Filed from this work:** #1074, an uploaded image can exhaust the backend's memory (already on main, in the ordinary upload routes). **Not built, to be filed:** a sweep for media directories left behind when the backend dies mid-import.
Author
Collaborator

Done: PR #1076 is merged (export schema 18). CI was green on the merged commit.

Across three PRs, a campaign's export and import now carry everything this issue listed:

  • #1053 (schema 17): quest details, character sheets with versions and kept rolls, parties.
  • #1072: the archive handled on disk within bounds; the largest import is an Admin setting.
  • #1076 (schema 18): handout attachment files, and maps with their markers, links, trail and original images.

How #1076 differs from the plan (details in the PR)

  • Map pictures are checked in the worker after the import answers.
  • A file over a per-file cap, a pixel bound or a count bound is left out and listed, not a refusal. The bounds: 8192×6144 for a map, 4000×3000 for an attachment or page picture, 200 map pictures and 5,000 attachments per import.
  • An independent review found eight defects before the merge; all are fixed and tested.

One CI failure on the way: the first run failed the session-hygiene test (#814), because the new post-commit look used the pooled session. Fixed with task_session.

Follow-ups, each its own issue

  • #1054: parse data.json in the worker.
  • #1069: Workbench generation runs (needs the owner's decision).
  • #1070: rule cards re-link to their pack.
  • #1074: the upload routes decode images with no pixel bound.
  • #1075: orphaned media directories are never swept.
  • #1077: map jobs share the general worker.

Reaches dev with the next prerelease (v4.5.0-rc4).

**Done: PR #1076 is merged (export schema 18). CI was green on the merged commit.** Across three PRs, a campaign's export and import now carry everything this issue listed: - **#1053** (schema 17): quest details, character sheets with versions and kept rolls, parties. - **#1072:** the archive handled on disk within bounds; the largest import is an Admin setting. - **#1076** (schema 18): handout attachment files, and maps with their markers, links, trail and original images. **How #1076 differs from the plan** (details in the PR) - Map pictures are checked in the worker after the import answers. - A file over a per-file cap, a pixel bound or a count bound is left out and listed, not a refusal. The bounds: 8192×6144 for a map, 4000×3000 for an attachment or page picture, 200 map pictures and 5,000 attachments per import. - An independent review found eight defects before the merge; all are fixed and tested. **One CI failure on the way:** the first run failed the session-hygiene test (#814), because the new post-commit look used the pooled session. Fixed with `task_session`. **Follow-ups, each its own issue** - #1054: parse `data.json` in the worker. - #1069: Workbench generation runs (needs the owner's decision). - #1070: rule cards re-link to their pack. - #1074: the upload routes decode images with no pixel bound. - #1075: orphaned media directories are never swept. - #1077: map jobs share the general worker. Reaches dev with the next prerelease (`v4.5.0-rc4`).
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/Quest-Board#991
No description provided.