A JPEG too large for the server is made smaller as it is stored, not refused (#1096) #1103
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/1096-shrink-large-pictures"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #1096 (D284, the owner's answer to Q156). Not to be merged before the independent review reports.
Summary
Image.draft), so the picture is never held at full size.Where this departs from the plan
There is no resize step, so a JPEG just over the bound comes out at a quarter of its pixels.
draftoffers only 1/2, 1/4 and 1/8. An intermediate "at or above the target" is therefore over the bound by up to four times, and that intermediate is what does not fit. Measured on a 24 MP intermediate: the decode alone is +94 to +96 MB, and every way of resizing it costs more (+97 to +211 MB).So the picture is decoded at the largest scale whose result is within the bound, and that is stored:
Before this, all but the first were refused.
Ceilings, from the header, before any decode
draftdoes not avoid the progressive coefficient buffer, which stays at full size. That is why a progressive JPEG gains so little.draftdoes not take, the picture is refused._decode_smallerchecks the image's size and decoder configuration before any load; there is no path from there to a full-size decode.A bug on main, found while measuring and fixed here
_jpeg_profilestopped at an escapedFF 00between segments, which Pillow and libjpeg step over. Two bytes after the start marker made a 12.5 MP progressive JPEG read as "baseline, 0 scans": it was counted once, missed the scan cap, and decoded at +124 MB in a 512 MB container. The walk now steps overFF 00and runs on the sanitised bytes the decoder receives. Its own fragment:changelog.d/1096-progressive-jpeg-escaped-ff.security.md.Measured
docker run -m 512m, the real validator, peak over the state before the call:The worst accepted file costs +62 MB, the same envelope as before. The slowest takes 1.4 s.
Through nginx on a warm 512 MB backend: the files above were stored at the sizes shown, upright, with no metadata; past-ceiling files answered 422 in under half a second. Anonymous memory peaked at 494 MB, when the worst file was sent three times straight after a batch that had left the backend at 429 MB.
oom_kill 0, no restarts.Known and unchanged
Changelog
changelog.d/1096-shrink-large-pictures.changed.mdchangelog.d/1096-progressive-jpeg-escaped-ff.security.mdTest plan
8052 passed, 14 skipped, 107 deselected.tests/wiki,tests/export, the damaged-image and session-hygiene tests:1790 passed.test_shrink_large_pictures.pyandtest_import_shrinks_pictures.py. Against main'simage_service:63 failed, 5 passed.🤖 Generated with Claude Code
Independent review of
36236146: not to be merged yet. The new shrink path is sound. Two bypasses were found in the code it builds on, both on main today, and both are being fixed on this branch.Findings
_strip_jpegreturns the file unstripped when it stops before a scan. Four bytes in front of the 66 KB MPF file from the tests tookImage.opento 1,650 MB. This defeats #1088; #1087 is reopened.decoderconfigcheck were removed.Found sound
draftpicks exactly the planned scale (300,000 random sizes, 0 mismatches), and the draft is applied to the open that is loaded.Not checked by the review: Pillow's C decoder and libjpeg's source (reasoned from memory); the pytest suites; memory at the real ceilings.
All five review findings fixed in
a87aa774(main merged in first). One more of the same shape was found and fixed in the GIF stripper._strip_jpegreturned the file unstripped when it stopped early_jpeg_header, steps from the start marker to the first scan as Pillow's_opendoes. It ends at a scan or raises; there is no other exit. The strip's only return is in the scan branchdecoderconfigcheck was untestedThe same pass over the other strippers
VP8Xchunk's flags are cleared, not the last. Harmless either way, since the chunks are dropped.Measured, at the header read
FF D8 FF D9in frontDRIor a one-lengthDQTNew tests
154 failed, 419 passed.Ran at
a87aa7748330 passed, 14 skipped, 107 deselected.tests/wiki,tests/export, the damaged-image and session-hygiene tests:2066 passed.oom_kill 0, no restarts.t1andt3against this commit. Both prefixed files are refused at 0.0 MB, and the as-built ones open as the 40×30 picture inside them.Not done
One figure corrected: the envelope's real top is +66 to +67 MB, not +62. The dearest file is one at the upload cap that also carries EXIF, because the strip holds a second copy during the decode. The progressive CMYK ceiling reaches +68 MB with such a file.