Export: a campaign with more than 32,767 pages, posts or markers cannot be exported #1113

New issue

Closed

opened 2026-10-05 22:02:57 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-10-05 22:02:57 +00:00

Collaborator

Found while building #1105 (PR #1112). Read from the code; not run.

What

The export collects a campaign's rows with IN (...) lists that carry one bind argument per parent row. asyncpg refuses a statement with more than 32,767 arguments ("query arguments cannot exceed 32767"), which the route would answer as a 500. #1105 fixed the one such statement on the import path (wiki_links.load_names); the export still has them:

quest_archive.py:125 and sheet_archive.py:502 (one per page)
files_archive.py:217 (one per page)
prep_shelf_archive.py:279, 313, 478, 479 (one per session, one per page)
discussion_archive.py:97, 106, 107 (one per thread, one per post)
atlas_archive.py:151, 164, 165, 177 (one per map, one per marker)
sheet_archive.py:524, party_archive.py:127

Why it matters now

#1105 sets what one import takes (D289): 50,000 wiki pages, 100,000 posts, 50,000 markers. A campaign imported at those sizes would then fail to export. Nothing real is near them (production's largest campaign has 62 pages), so this is a correctness gap, not something anyone has met.

Intended approach

Replace each list with a join or subquery on the campaign, as #1105 did for load_names, or chunk it. A test that exports a campaign through a statement spy and asserts that no statement carries more than a few hundred binds, as test_import_bounds.py does for import. Then check the same pattern outside export (campaign purge, the wiki's own pages).

Found while building #1105 (PR #1112). Read from the code; not run. ## What The export collects a campaign's rows with `IN (...)` lists that carry one bind argument per parent row. asyncpg refuses a statement with more than 32,767 arguments ("query arguments cannot exceed 32767"), which the route would answer as a 500. #1105 fixed the one such statement on the import path (`wiki_links.load_names`); the export still has them: - `quest_archive.py:125` and `sheet_archive.py:502` (one per page) - `files_archive.py:217` (one per page) - `prep_shelf_archive.py:279, 313, 478, 479` (one per session, one per page) - `discussion_archive.py:97, 106, 107` (one per thread, one per post) - `atlas_archive.py:151, 164, 165, 177` (one per map, one per marker) - `sheet_archive.py:524`, `party_archive.py:127` ## Why it matters now #1105 sets what one import takes (D289): 50,000 wiki pages, 100,000 posts, 50,000 markers. A campaign imported at those sizes would then fail to export. Nothing real is near them (production's largest campaign has 62 pages), so this is a correctness gap, not something anyone has met. ## Intended approach Replace each list with a join or subquery on the campaign, as #1105 did for `load_names`, or chunk it. A test that exports a campaign through a statement spy and asserts that no statement carries more than a few hundred binds, as `test_import_bounds.py` does for import. Then check the same pattern outside export (campaign purge, the wiki's own pages).

claude-bot referenced this issue

2026-10-05 22:02:59 +00:00

An import can build far more than it parses: bound it by row counts #1105

claude-bot referenced this issue

2026-10-05 23:01:35 +00:00

An import can build far more than it parses: bound it by row counts #1105