An import can build far more than it parses: bound it by row counts

claude-bot commented

2026-10-05 10:14:39 +00:00

Collaborator

Found while building #1054 (PR #1104). It is true on main today, at the old caps.

What

An import's data.json is capped by its size and by a count of its values. Neither bounds what the import then builds: a small file of very many short rows costs far more as rows than as JSON.

Measured on main's code, with files inside the caps of 8 MB and 400,000 values:

File	Result
80,000 two-field sessions (2.2 MB)	a 1 GB container to 946 MB after about 35 minutes
44,000 pages	536 MB, then a 500: asyncpg "query arguments cannot exceed 32767"
57,000 members	423 MB, imported

Any of these would have killed a 512 MB backend.

Where it stands after PR #1104

The import runs in the worker under a memory bound: once the process has grown 400 MB the import is stopped with a 422, "This file holds more than this server can read in one import." That makes it safe. It is not a good answer:

the file is refused only after tens of seconds of work;
the limit is in megabytes of the server's memory, which no one can check a file against;
the 500 on about 44,000 pages is a separate bug the bound happens to mask.

Intended approach

Count rows before building anything, from the parsed data: sessions, pages, members, notes, transcript segments, markers, shelf cards and the rest. Refuse over a bound per kind, in words that name the kind and the bound.
Choose the bounds from real use, generously. Production's largest campaign has 21 sessions and 62 pages. A bound of a few thousand sessions and tens of thousands of pages refuses nothing real. Then measure what a file at every bound costs, and keep the memory stop as the backstop.
Fix the 32767-argument 500: batch the statement that carries one argument per row.
Session numbers are allocated one query each, which makes a long session list slow before it is large. Allocate them in one pass.
Tests: a file at each bound imports; one over is refused before any row is built; the 44,000-page case no longer answers 500.

The bounds are limits a person could meet, so they go to the owner as a question before they are set.

#1054, #1104, #1072.

Found while building #1054 (PR #1104). It is true on main today, at the old caps. ## What An import's `data.json` is capped by its size and by a count of its values. Neither bounds what the import then builds: a small file of very many short rows costs far more as rows than as JSON. Measured on main's code, with files inside the caps of 8 MB and 400,000 values: | File | Result | |---|---| | 80,000 two-field sessions (2.2 MB) | a 1 GB container to 946 MB after about 35 minutes | | 44,000 pages | 536 MB, then a 500: asyncpg "query arguments cannot exceed 32767" | | 57,000 members | 423 MB, imported | Any of these would have killed a 512 MB backend. ## Where it stands after PR #1104 The import runs in the worker under a memory bound: once the process has grown 400 MB the import is stopped with a 422, "This file holds more than this server can read in one import." That makes it safe. It is not a good answer: - the file is refused only after tens of seconds of work; - the limit is in megabytes of the server's memory, which no one can check a file against; - the 500 on about 44,000 pages is a separate bug the bound happens to mask. ## Intended approach - **Count rows before building anything**, from the parsed data: sessions, pages, members, notes, transcript segments, markers, shelf cards and the rest. Refuse over a bound per kind, in words that name the kind and the bound. - **Choose the bounds from real use, generously.** Production's largest campaign has 21 sessions and 62 pages. A bound of a few thousand sessions and tens of thousands of pages refuses nothing real. Then measure what a file at every bound costs, and keep the memory stop as the backstop. - **Fix the 32767-argument 500**: batch the statement that carries one argument per row. - **Session numbers are allocated one query each**, which makes a long session list slow before it is large. Allocate them in one pass. - Tests: a file at each bound imports; one over is refused before any row is built; the 44,000-page case no longer answers 500. The bounds are limits a person could meet, so they go to the owner as a question before they are set. ## Related #1054, #1104, #1072.

claude-bot referenced this issue

2026-10-05 11:58:37 +00:00

[Security] Campaign import reads data.json with no cap on its decompressed size #1054

claude-bot commented

2026-10-05 19:19:26 +00:00

Author

Collaborator

Picking this up. The owner answered Q162 on 2026-10-05 with (a): 5,000 sessions, 50,000 wiki pages, 500 members, 500,000 transcript lines, and the same order for the smaller things. It will be recorded as D289.

Building on feat/1105-import-row-bounds: count every kind before anything is built, refuse in words that name the kind and the bound, batch the statements behind the 32767-argument 500, allocate session numbers in one pass, and measure a file at every bound against the 400 MB stop. If a bound can't be honoured inside the worker's memory, it goes back to the owner rather than being lowered quietly.

Picking this up. The owner answered Q162 on 2026-10-05 with (a): **5,000 sessions, 50,000 wiki pages, 500 members, 500,000 transcript lines, and the same order for the smaller things.** It will be recorded as D289. Building on `feat/1105-import-row-bounds`: count every kind before anything is built, refuse in words that name the kind and the bound, batch the statements behind the 32767-argument 500, allocate session numbers in one pass, and measure a file at every bound against the 400 MB stop. If a bound can't be honoured inside the worker's memory, it goes back to the owner rather than being lowered quietly.

claude-bot referenced this issue

2026-10-05 19:19:26 +00:00

A picture too large for the server is shrunk on the way in, not refused #1096

rbrooks referenced this issue from a commit

2026-10-05 22:00:25 +00:00

feat(webapp): an import counts what a file holds before it builds anything (#1105)

claude-bot referenced this issue from a pull request that will close it,

2026-10-05 22:02:10 +00:00

feat(webapp): an import counts what a file holds before it builds anything (#1105) #1112

rbrooks referenced this issue from a commit

2026-10-05 22:02:10 +00:00

docs(design): D288 and D289 carry their question codes (#1105)

claude-bot referenced this issue

2026-10-05 22:02:57 +00:00

Export: a campaign with more than 32,767 pages, posts or markers cannot be exported #1113

claude-bot commented

2026-10-05 22:02:59 +00:00

Author

Collaborator

Built on feat/1105-import-row-bounds; PR #1112 is open and in CI, with an independent read-only review running before it merges.

Notes against this issue's plan:

All four of the owner's bounds fit: the four at once (a 23.2 MB file) grew the worker 230 MB in 34 s; the dearest in-bounds file built reached 342 MB of the 400 MB stop.
Transcript lines are text lines of sessions[].transcript: transcript segments never travel in an export, so there are no segment rows to count.
The 32767-argument 500 was wiki_links.load_names; fixed with a join. The export has the same pattern in many places: #1113.
Also found and fixed: page indexing on import was quadratic (10,000 pages took 103 s; 50,000 now take about 30 s).
The builder could not reach this issue from its sandbox, so it worked from my brief; I have checked the PR against the plan here.

Built on `feat/1105-import-row-bounds`; PR #1112 is open and in CI, with an independent read-only review running before it merges. Notes against this issue's plan: - All four of the owner's bounds fit: the four at once (a 23.2 MB file) grew the worker 230 MB in 34 s; the dearest in-bounds file built reached 342 MB of the 400 MB stop. - **Transcript lines** are text lines of `sessions[].transcript`: transcript segments never travel in an export, so there are no segment rows to count. - The 32767-argument 500 was `wiki_links.load_names`; fixed with a join. The export has the same pattern in many places: #1113. - Also found and fixed: page indexing on import was quadratic (10,000 pages took 103 s; 50,000 now take about 30 s). - The builder could not reach this issue from its sandbox, so it worked from my brief; I have checked the PR against the plan here.

rbrooks referenced this issue from a commit

2026-10-05 22:10:31 +00:00

docs(import): say which kinds are written as the import goes (#1105)

claude-bot closed this issue

2026-10-05 23:01:30 +00:00

claude-bot referenced this issue from a commit

2026-10-05 23:01:31 +00:00

Merge PR #1112: an import counts what a file holds before it builds anything (#1105)

claude-bot commented

2026-10-05 23:01:35 +00:00

Author

Collaborator

Done: PR #1112 merged with CI green on its final commit (14 checks), after an independent read-only review that found nothing blocking (its findings are on the PR). The owner's four bounds are in place (D289), with the rest of the table in docs/OPERATIONS.md. Follow-ups: #1113 (the export's one-bind-per-row lists), and the review's note that plan-built kinds (markers, attachments, party members, quest pages) are counted after their plans exist. Reaches dev with v4.5.0-rc9.

claude-bot referenced this issue

2026-10-05 23:01:41 +00:00

chore(release): v4.5.0-rc9 #1114

Rows
Columns

An import can build far more than it parses: bound it by row counts #1105

What

Where it stands after PR #1104

Intended approach

Related