An upgraded install's media volume stays root-owned: the backend cannot write uploads (production is in this state) #1064

New issue

Closed

opened 2026-10-04 03:38:10 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-10-04 03:38:10 +00:00

Collaborator

Found on 2026-10-04 while checking why a map upload returned 500.

Seen

#914 made the image create /app/media owned by appuser, so a fresh media volume is writable. Docker only copies an image's ownership into a volume when the volume is first created, and #914's own commit says existing volumes are unaffected. An install whose media volume predates #914 keeps it root-owned.

Checked on both hosts, as the backend's own user (uid 999):

Host	`/app/media` owner	Write test
dev (`docker-test`, `v4.5.0-rc2`)	`999:999`	writable
prod (`docker-host`, 4.4.1 images)	`0:0`	`Permission denied`

#914 is not in v4.4.1. On prod today the backend cannot write anything under /app/media; the directory is empty. After a 4.5.0 deploy, every upload there (page pictures, handout files, map images and their tiles, the map worker's output) would fail with a PermissionError, surfacing as a 500.

Any self-hoster upgrading from before #914 is in the same state.

Fix

The migrate service already runs as root before the app starts, to prepare ./backups (#414). Give it the media volume too and have it make /app/media owned by appuser when it is not. That is one cheap check of the top-level owner, with a recursive chown only when it is wrong, so a large tile tree is not walked on every start. An existing install is then repaired by the next docker compose up -d, with nothing for the operator to remember.

Prove it on a throwaway stack: a volume created root-owned, then up, then a write as appuser.

Production

Nothing was changed on prod; the write test created no file. The fix reaches prod with 4.5.0. Until then uploads that write under /app/media fail there. A one-off docker compose exec -u root backend chown appuser:appuser /app/media would repair it now; that is the owner's call.

Found on 2026-10-04 while checking why a map upload returned 500. ## Seen #914 made the image create `/app/media` owned by `appuser`, so a **fresh** `media` volume is writable. Docker only copies an image's ownership into a volume when the volume is first created, and #914's own commit says existing volumes are unaffected. An install whose `media` volume predates #914 keeps it root-owned. Checked on both hosts, as the backend's own user (uid 999): | Host | `/app/media` owner | Write test | |---|---|---| | dev (`docker-test`, `v4.5.0-rc2`) | `999:999` | writable | | **prod** (`docker-host`, 4.4.1 images) | **`0:0`** | **`Permission denied`** | #914 is not in `v4.4.1`. On prod today the backend cannot write anything under `/app/media`; the directory is empty. After a 4.5.0 deploy, every upload there (page pictures, handout files, map images and their tiles, the map worker's output) would fail with a `PermissionError`, surfacing as a 500. Any self-hoster upgrading from before #914 is in the same state. ## Fix The `migrate` service already runs as root before the app starts, to prepare `./backups` (#414). Give it the `media` volume too and have it make `/app/media` owned by `appuser` when it is not. That is one cheap check of the top-level owner, with a recursive chown only when it is wrong, so a large tile tree is not walked on every start. An existing install is then repaired by the next `docker compose up -d`, with nothing for the operator to remember. Prove it on a throwaway stack: a volume created root-owned, then `up`, then a write as `appuser`. ## Production Nothing was changed on prod; the write test created no file. The fix reaches prod with 4.5.0. Until then uploads that write under `/app/media` fail there. A one-off `docker compose exec -u root backend chown appuser:appuser /app/media` would repair it now; that is the owner's call.

rbrooks referenced this issue from a commit

2026-10-04 03:49:05 +00:00

fix: migrate repairs a root-owned media volume (#1064)

claude-bot referenced this issue from a pull request that will close it,

2026-10-04 03:49:47 +00:00

An upgrade repairs a root-owned media volume by itself (#1064) #1068

claude-bot commented

2026-10-04 03:49:50 +00:00

Author

Collaborator

A correction to the report above, from proving the fix (the fix is PR #1068).

An empty root-owned volume does not reproduce the failure. Docker re-initialises an empty named volume from the image's directory when a container mounts it, ownership included. With an image that has #914, an empty volume comes back owned by appuser unaided. The bug needs a volume that already holds files and is root-owned.

What that means for production:

Today (4.4.1 images, before #914): the volume is root-owned and the backend cannot write to it. That part of the report stands.
After 4.5.0: prod's volume is root-owned but empty, so the 4.5.0 image should initialise it correctly by itself. The risk I described for prod after the upgrade was overstated.
The fix is still right: it covers any install whose root-owned volume is not empty, and it makes the outcome certain either way: one stat per start, and a recursive chown only when the owner is wrong.

The one-off chown on prod is only worth doing if uploads on 4.4.1 matter before 4.5.0 ships.

**A correction to the report above, from proving the fix (the fix is PR #1068).** An *empty* root-owned volume does not reproduce the failure. Docker re-initialises an empty named volume from the image's directory when a container mounts it, ownership included. With an image that has #914, an empty volume comes back owned by `appuser` unaided. The bug needs a volume that already **holds files** and is root-owned. What that means for production: - **Today (4.4.1 images, before #914):** the volume is root-owned and the backend cannot write to it. That part of the report stands. - **After 4.5.0:** prod's volume is root-owned but **empty**, so the 4.5.0 image should initialise it correctly by itself. The risk I described for prod after the upgrade was overstated. - **The fix is still right:** it covers any install whose root-owned volume is not empty, and it makes the outcome certain either way: one `stat` per start, and a recursive chown only when the owner is wrong. The one-off `chown` on prod is only worth doing if uploads on 4.4.1 matter before 4.5.0 ships.

claude-bot closed this issue

2026-10-04 04:20:14 +00:00

claude-bot referenced this issue from a commit

2026-10-04 04:20:15 +00:00

Merge PR #1068: an upgrade repairs a root-owned media volume by itself (#1064)

claude-bot commented

2026-10-04 04:20:27 +00:00

Author

Collaborator

Fixed in PR #1068, merged to main. CI was green on the PR's head.

The change: the migrate service mounts media and, when the top-level owner is not appuser, logs migrate: /app/media is not owned by appuser; repairing (#1064) and runs a recursive chown. Otherwise it costs one stat per start.
Proved on docker-test in a throwaway project: a non-empty root-owned volume is repaired on up and on the migrate-safe path; a correct volume is not walked; a fresh volume is a no-op.
Ships in: the next prerelease (v4.5.0-rc3) and 4.5.0.

Still open, for the owner: production runs 4.4.1, where the volume is root-owned and uploads fail today. Either run the one-off docker compose exec -u root backend chown appuser:appuser /app/media now, or wait for 4.5.0. Nothing has been changed on production.

**Fixed in PR #1068, merged to main. CI was green on the PR's head.** - **The change:** the `migrate` service mounts `media` and, when the top-level owner is not `appuser`, logs `migrate: /app/media is not owned by appuser; repairing (#1064)` and runs a recursive chown. Otherwise it costs one `stat` per start. - **Proved on docker-test** in a throwaway project: a non-empty root-owned volume is repaired on `up` and on the `migrate-safe` path; a correct volume is not walked; a fresh volume is a no-op. - **Ships in:** the next prerelease (`v4.5.0-rc3`) and 4.5.0. **Still open, for the owner:** production runs 4.4.1, where the volume is root-owned and uploads fail today. Either run the one-off `docker compose exec -u root backend chown appuser:appuser /app/media` now, or wait for 4.5.0. Nothing has been changed on production.

claude-bot referenced this issue

2026-10-05 05:57:33 +00:00

Decisions D279 to D287, the owner's answers of 5 October (Q151 to Q159), with two of them built (#677, #367) #1097