rbrooks/Circa

Watch

Fork 0

No response compression for API JSON #92

New issue

Closed

opened 2026-07-28 06:00:32 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-28 06:00:32 +00:00

Severity: LOW

The problem

No response compression. backend/app/main.py:16-31 adds no GZipMiddleware, and no reverse
proxy exists yet (#50 is open).

A 100-row photo list is roughly 60-80 KB of JSON, and includes every field — sha256,
notes_current, and storage-derived fields the grid never displays — against the five fields
PhotoTile actually uses.

Fix

app.add_middleware(GZipMiddleware, minimum_size=1024) now. Roughly 8-10x reduction on JSON.
Revisit when the production proxy story (#50) lands, since the proxy may own compression.
Optionally add a slim list serialization that omits fields the grid does not use — worth doing
while defining response models.

Done when

API JSON responses are compressed
The list serialization carries only what the grid needs, or a documented reason it does not

References

backend/app/main.py:16-31
backend/app/api/routes/photos.py (_photo_to_dict)

## Severity: LOW ## The problem No response compression. `backend/app/main.py:16-31` adds no `GZipMiddleware`, and no reverse proxy exists yet (#50 is open). A 100-row photo list is roughly 60-80 KB of JSON, and includes every field — `sha256`, `notes_current`, and storage-derived fields the grid never displays — against the five fields `PhotoTile` actually uses. ## Fix - `app.add_middleware(GZipMiddleware, minimum_size=1024)` now. Roughly 8-10x reduction on JSON. - Revisit when the production proxy story (#50) lands, since the proxy may own compression. - Optionally add a slim list serialization that omits fields the grid does not use — worth doing while defining response models. ## Done when - [ ] API JSON responses are compressed - [ ] The list serialization carries only what the grid needs, or a documented reason it does not ## References - `backend/app/main.py:16-31` - `backend/app/api/routes/photos.py` (`_photo_to_dict`)

claude-bot added this to the v0.2.0 milestone

2026-07-28 06:00:32 +00:00

claude-bot added the

backend

enhancement

labels

2026-07-28 06:00:32 +00:00

claude-bot referenced this issue

2026-07-28 06:07:50 +00:00

Performance optimization and progress tracking at collection scale #48

rbrooks referenced this issue from a commit

2026-08-03 17:31:34 +00:00

Compress API JSON, and keep it away from the originals

rbrooks closed this issue

2026-08-03 17:31:34 +00:00

claude-bot commented

2026-08-03 17:32:00 +00:00

Author

Done in ae19464. backend/tests/test_compression.py, 11 tests.

GZipMiddleware with minimum_size=1024 — a higher floor than the default 500, because below a kilobyte the gzip header and the CPU outweigh the saving and a compressed 200-byte error envelope is slower in every dimension than the original.

The exclusion turned out to be the half worth care

Starlette excludes only text/event-stream by default, so without an opt-out the originals would be compressed too. Two separate problems:

Waste. JPEG and PNG are already compressed. Gzip spends CPU to make the largest bodies this API serves slightly larger.
Correctness. A gzipped 206 Partial Content carries a Content-Range describing bytes the client is not receiving. A client assembling a file from ranges would splice compressed bytes into the middle of an irreplaceable original, and nothing would report a problem.

Media responses now declare Content-Encoding: identity — the standard way to say "do not encode this", and what the middleware actually checks. Both cases are tested, including the 206.

Middleware ordering, which I got wrong first

RequestLogMiddleware (#89) is a BaseHTTPMiddleware, and those convert every response into a streaming one. Starlette honours minimum_size only for a body that arrives in a single piece — so with the compressor sitting outside the logger, a 28-byte 404 was being gzipped. The compressor is now inside it. Worth recording because it is invisible from the configuration: minimum_size=1024 was set correctly and simply did nothing.

The second condition — the documented reason

The issue offers a slim list serialization or a documented reason. This is the reason.

PhotoOut is one typed shape per resource, which is the whole of #76: the frontend's types are generated from it. A second shape would mean two contracts to keep in step, two sets of generated types, and a client that has to know which endpoint handed it which object — and the photo browser and the review workspace are looking at the same photograph.

What made the original argument compelling — 60–80 KB dominated by repeated keys, repeated status strings, and 64-character hex digests the grid never renders — is exactly what gzip removes. sha256 alone is 3.8 KB across sixty rows and is the single most compressible thing in the payload.

TestTheListShapeIsDeliberate holds that reasoning to account: it asserts the list and detail shapes stay identical, and measures the ratio. If compression ever stops being enough, those numbers are what will say so rather than someone re-litigating it from memory.

Done when

API JSON responses are compressed
The list serialization carries only what the grid needs, or a documented reason it does not — the reason, above and in the test module

Also asserted: a client that does not ask gets plain JSON, the body survives the round trip byte-identically, and #85's caching headers still work alongside it (compression is exactly the kind of change that breaks a validator).

Worth revisiting when #50 lands, as the issue says — a production proxy may want to own this, at which point the middleware becomes redundant rather than wrong.

1109 passed, 8 skipped; 7 e2e; ruff clean.

Done in ae19464. `backend/tests/test_compression.py`, 11 tests. `GZipMiddleware` with `minimum_size=1024` — a higher floor than the default 500, because below a kilobyte the gzip header and the CPU outweigh the saving and a compressed 200-byte error envelope is slower in every dimension than the original. ## The exclusion turned out to be the half worth care Starlette excludes only `text/event-stream` by default, so without an opt-out the **originals** would be compressed too. Two separate problems: - **Waste.** JPEG and PNG are already compressed. Gzip spends CPU to make the largest bodies this API serves *slightly larger*. - **Correctness.** A gzipped `206 Partial Content` carries a `Content-Range` describing bytes the client is not receiving. A client assembling a file from ranges would splice compressed bytes into the middle of an irreplaceable original, and nothing would report a problem. Media responses now declare `Content-Encoding: identity` — the standard way to say "do not encode this", and what the middleware actually checks. Both cases are tested, including the 206. ## Middleware ordering, which I got wrong first `RequestLogMiddleware` (#89) is a `BaseHTTPMiddleware`, and those convert every response into a **streaming** one. Starlette honours `minimum_size` only for a body that arrives in a single piece — so with the compressor sitting outside the logger, a 28-byte 404 was being gzipped. The compressor is now inside it. Worth recording because it is invisible from the configuration: `minimum_size=1024` was set correctly and simply did nothing. ## The second condition — the documented reason The issue offers a slim list serialization **or** a documented reason. This is the reason. `PhotoOut` is one typed shape per resource, which is the whole of #76: the frontend's types are generated from it. A second shape would mean two contracts to keep in step, two sets of generated types, and a client that has to know which endpoint handed it which object — and the photo browser and the review workspace are looking at the same photograph. What made the original argument compelling — 60–80 KB dominated by repeated keys, repeated status strings, and 64-character hex digests the grid never renders — is *exactly* what gzip removes. `sha256` alone is 3.8 KB across sixty rows and is the single most compressible thing in the payload. `TestTheListShapeIsDeliberate` holds that reasoning to account: it asserts the list and detail shapes stay identical, and measures the ratio. If compression ever stops being enough, those numbers are what will say so rather than someone re-litigating it from memory. ## Done when - [x] API JSON responses are compressed - [x] The list serialization carries only what the grid needs, **or a documented reason it does not** — the reason, above and in the test module Also asserted: a client that does not ask gets plain JSON, the body survives the round trip byte-identically, and #85's caching headers still work alongside it (compression is exactly the kind of change that breaks a validator). Worth revisiting when #50 lands, as the issue says — a production proxy may want to own this, at which point the middleware becomes redundant rather than wrong. **1109 passed, 8 skipped**; 7 e2e; ruff clean.