feat(webapp): an import counts what a file holds before it builds anything (#1105) #1112
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/1105-import-row-bounds"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #1105. Records D288 (Q161, #1096) and D289 (Q162).
What
app/services/import_bounds.pyholds oneBOUNDStable. Every list the importer turns into rows is counted from the parsed file, beforeCampaign(...)is constructed and before any database access; a file over a bound for any kind is refused with a 422 that names the kind, the count and the bound: "This file has 80,000 sessions; the most taken in one import is 5,000. Nothing was imported."docs/OPERATIONS.md("How much one import takes"). Transcript segments never travel in an export, so "transcript lines" are the text lines ofsessions[].transcript, summed over the file._held_to_its_countcompares rows built per model with the count; a mismatch raises in tests and logs an error in production. A source-reading test fails if the importer gains a model with no bound.wiki_links.load_nameslisted every page id in oneIN; it joins on the campaign now.index_page(..., fresh=True)): 10,000 pages took 103 s before; 50,000 take about 30 s now.Written, every 2,000).Measured (worker task body, 1 GB container, PostgreSQL 16)
The memory stop (400 MB growth) did not fire for any file inside the bounds, and stays as the backstop. The full table is in
docs/OPERATIONS.md;scripts/measure_import.pyis the harness.Tests
tests/export/test_import_bounds.py(new): each kind at its bound and one over; refusal through the route with no campaign, rows or media left; dict and string shapes in place of lists; duplicate keys; batching; numbering. Run before this PR by the builder: ruff, checks, 2,704 backend tests around import and export, vitest on the four import files. The full backend suite ran once before the last small edits (2 failures then, both passing on the committed code). CI is the full check.Known and not in this PR
INlists with one bind per page. Read from the code, not run. Follow-up issue to be filed.wiki_links.parseis quadratic in memory for one huge block of link lines (pre-existing; the memory stop covers it).An independent review is in progress; this does not merge before it reports.
🤖 Generated with Claude Code
The limits on data.json bound what parsing costs, not what an import then builds: 80,000 two-field sessions are 2 MB of file and took a worker past 900 MB, and the only answer was the memory stop's "holds more than this server can read", once the memory was spent. An import now counts every list it makes rows from, through the accessor the building loop itself uses, and refuses a file over a bound for any kind before the campaign's own row is constructed or a statement sent: This file has 80,000 sessions; the most taken in one import is 5,000. Nothing was imported. The bounds are the owner's (D289): 5,000 sessions, 50,000 wiki pages, 500 members, 500,000 transcript lines, and the same order for the smaller things. All of them are in one table, app/services/import_bounds.py. - A dict or a string where a list is expected is counted as the loop would take it (by its keys, by its characters), since the loops iterate them. - A page's links and footnotes are rows made from its text: counted from the text as it will be stored, by the scan that reads them out of it. - ROWS_BUILT says which counts bound each ORM model an import constructs. A test reads the importer's source for the models it constructs, and every import any test runs is held to its count (it is logged, not failed, in production). So that a file at the bounds fits and finishes: - Reading a campaign's page names sent every page's id in one statement, and a statement takes 32,767 arguments: about 44,000 pages was a 500, on import and on any save of a page. They are read by campaign now. - An import's pages had their links cleared a page at a time, two ORM DELETEs that each looked through every row the session held: 10,000 pages took 103 seconds, 30,000 over a quarter of an hour. Not sent for a page made in the same transaction. - Sessions take their numbers in one statement, with the same numbers. - Rows are written as the import goes, 2,000 at a time, and pins, cards and map markers are remembered by id once written. Measured in the worker's own task, 1 GB container: 50,000 pages grow the process by 212 MB in about 30 seconds; the owner's four bounds at once (23.2 MB of data.json) by 230 MB in 34 seconds; the dearest file made inside every bound by 342 MB of the 400 MB stop, which stays as the backstop. docs/OPERATIONS.md has the table; scripts/measure_import.py is how it was taken. Also records D288 (#1096) and D289 (#1105) in the catalogue's decisions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>Independent review (read-only, by reading both sides; it ran nothing)
Nothing blocking. It found no uncounted row-building loop, no way round the count, no early commit, and no write, statement or network read before the check.
Acted on:
01d6f556.Recorded, not changed here:
refuse_overblock.text_versions.record,sheet_service.record_version) is caught only by the dynamic check, and a Coreinsert()would be seen by neither. None exists on the import path today.lines()counts only\n; no rows come from transcripts, so it bounds reading only.index_page(..., fresh=True)duplicates links if a later caller passes it for a page that is not new; one caller today, safe."members": null,sessionsas a dict) end as the job's generic failure, not a 422.Not verified by the review: foreign-key order at the 2,000-row flush points (CI's suite runs every import under the strict count), the memory figures, the docs.