rbrooks/Quest-Board

Watch

Fork 0

feat(backend): content packs foundation: tables, import pipeline, search, export, endpoints (#553) #557

Merged

claude-bot merged 3 commits from feat/553-content-packs-foundation into main

2026-09-08 07:02:15 +00:00

claude-bot commented

2026-09-08 06:45:35 +00:00

Contributor

Lane 1 of the content packs feature (#553), built to docs/design/content-packs-spec.md. Backend only.

Tables and migration. content_packs and reference_entries as specified: two scopes, per-pack licence and affirmation records, status, a generated tsvector with weighted fields, the partial unique indexes, GIN and prefix indexes. Migration 1c2d3e4f5a6b chained on 0b1c2d3e4f5a, raw SQL, one head, applied end to end on a clean database; a test pins the head count and that the model's search expression matches the migration's byte for byte (the test harness uses create_all, so drift would otherwise be invisible).

Pipeline. Upload or fetch by URL (30 s, 50 MiB streamed, three redirects, every hop refused if it resolves to a private or loopback address, content-type checked); the affirmation enforced server-side with the text returned in the 422 and the version, hash, user and time recorded on the pack; format detection through a registry with a fixed detection order; the native adapter (.qbpack, a bare JSON array, CSV); sanitisation (nh3 then markdownify), caps on summary, body and record count; stats validated against the system's schema with warnings rather than rejections; slug de-duplication; one transaction; audit rows; packs over 2 MiB through a Celery task with status polling.

Search. Name-prefix first, then weighted full-text rank, campaign packs above instance packs for the same kind and slug (ranked per group so a differently titled campaign copy still wins), GM-only records only for the campaign's GM, disabled packs excluded.

Endpoints as specified for members, admins (who may act on any scope) and GMs, one router; starters declared before the id route. Export writes a native zip and withholds non-redistributable records with reasons.

Docs: docs/CONTENT-PACKS.md (what a pack is, the affirmation, the three native shapes with examples, export), docs/API.md, one CHANGELOG entry naming the migration.

Deviations, stated in the code: system.json in a pack creates a game system but never updates an existing one (a schema is shared by every campaign; an uploaded file should not redefine it); pack assets are recorded but not stored in phase 1; a GM searching without naming a campaign sees members-only records.

Verification: 2,634 passed, 13 skipped (118 new across 7 files); ruff check and format clean.

Adapter lanes (Foundry; 5etools and orcbrew; Open5e and starter scripts) follow as separate PRs that register with this registry.

🤖 Generated with Claude Code

Lane 1 of the content packs feature (#553), built to `docs/design/content-packs-spec.md`. Backend only. **Tables and migration.** `content_packs` and `reference_entries` as specified: two scopes, per-pack licence and affirmation records, status, a generated `tsvector` with weighted fields, the partial unique indexes, GIN and prefix indexes. Migration `1c2d3e4f5a6b` chained on `0b1c2d3e4f5a`, raw SQL, one head, applied end to end on a clean database; a test pins the head count and that the model's search expression matches the migration's byte for byte (the test harness uses `create_all`, so drift would otherwise be invisible). **Pipeline.** Upload or fetch by URL (30 s, 50 MiB streamed, three redirects, every hop refused if it resolves to a private or loopback address, content-type checked); the affirmation enforced server-side with the text returned in the 422 and the version, hash, user and time recorded on the pack; format detection through a registry with a fixed detection order; the native adapter (`.qbpack`, a bare JSON array, CSV); sanitisation (nh3 then markdownify), caps on summary, body and record count; stats validated against the system's schema with warnings rather than rejections; slug de-duplication; one transaction; audit rows; packs over 2 MiB through a Celery task with status polling. **Search.** Name-prefix first, then weighted full-text rank, campaign packs above instance packs for the same kind and slug (ranked per group so a differently titled campaign copy still wins), GM-only records only for the campaign's GM, disabled packs excluded. **Endpoints** as specified for members, admins (who may act on any scope) and GMs, one router; `starters` declared before the id route. **Export** writes a native zip and withholds non-redistributable records with reasons. **Docs:** `docs/CONTENT-PACKS.md` (what a pack is, the affirmation, the three native shapes with examples, export), `docs/API.md`, one CHANGELOG entry naming the migration. **Deviations, stated in the code:** `system.json` in a pack creates a game system but never updates an existing one (a schema is shared by every campaign; an uploaded file should not redefine it); pack assets are recorded but not stored in phase 1; a GM searching without naming a campaign sees members-only records. **Verification:** 2,634 passed, 13 skipped (118 new across 7 files); ruff check and format clean. Adapter lanes (Foundry; 5etools and orcbrew; Open5e and starter scripts) follow as separate PRs that register with this registry. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 3 commits

2026-09-08 06:45:35 +00:00

feat(backend): the content_packs and reference_entries tables (#553 ) 448d1a48a4

The storage half of imported rules and bestiary lookup. A content pack is a
unit of provenance — one source, one licence declaration, one affirmation by
one person, one scope — because disable and remove have to be one action with
a blast radius an operator can state when a rightsholder writes in (#555).
Reference entries are the normalised records inside it: a creature from a
Foundry export and a house rule typed into a spreadsheet are the same row with
a different `kind`.

Three choices worth the reading time:

* **Two partial unique indexes, not one composite constraint.** `UNIQUE
  (campaign_id, key)` does not constrain instance packs at all, because NULLs
  are distinct in a unique index — the same instance key could be loaded any
  number of times. `(key) WHERE scope = 'instance'` and `(campaign_id, key)
  WHERE scope = 'campaign'` say what is actually meant.
* **TEXT + CHECK, never `sa.Enum`**, per the house rule and the
  `game_system_schemas.status` precedent. A fifth status should cost a
  constraint alteration, not a type rewrite. The migration is raw SQL for the
  same reason the constraint is: it is what a reviewer has to be able to read.
* **`traits` is cast to text inside the generated `search_vector`.** A
  generated column's expression may not contain a subquery, so the JSON array
  cannot be unnested; the punctuation tokenises away, and `["undead","fire"]`
  indexes as `undead` and `fire`. Verified against Postgres 16 — the cast is
  immutable enough for a STORED column, which was the open question.

`system_id` is denormalised onto each entry from its pack. Search filters by
game system on every query, and an index cannot span a join; packs never change
system, so the copy cannot drift.

The timestamp columns carry Python-side defaults as well as their SQL ones. A
server default is not fetched back after an INSERT and a server-side `onupdate`
*expires* the attribute after an UPDATE, so building the API response from a
freshly written row would trigger a lazy refresh — which in an async session is
a `MissingGreenlet`, not a query.

Adds `nh3` and `markdownify`, used by the sanitiser in the following commit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(backend): import, search and export for content packs (#553 ) 6ac6260667

The working half: `app/reference/` reads a pack, normalises it, stores it,
finds it again, and hands back only what may be handed back.

**`adapters/base.py` is the frozen contract** — the normalised record and the
`Adapter` protocol, verbatim from the spec, because lanes 2-4 build the
Foundry, 5etools, orcbrew and Open5e readers against it and a contract that
moves under them is not a contract. Registering one is two lines in
`adapters/__init__.py`; detection order is declared separately from
registration order so appending at the bottom of that file cannot change which
adapter claims a file.

**`questboard.py`** reads all three native shapes: a `.qbpack` zip, a bare JSON
array whose key comes from the filename, and a CSV somebody wrote in a
spreadsheet. It deliberately does *not* claim a plain `.zip` — it is asked
first, and a zip is at least as likely to be a Foundry export.

**Sanitising is two paths, and the second one is the point.** Text containing
tags goes through an nh3 allowlist (script, style and iframe lose their
contents, not just their tags) and then markdownify. Text with no tags is left
alone, because measured, markdownify escapes `**bold**` to `\*\*bold\*\*` and
eats the blank lines between Markdown paragraphs — running everything through
it would corrupt every hand-written pack in the name of safety it did not need.
`javascript:` and `data:` link targets are scrubbed on both paths, since nh3
cannot see a Markdown link.

**Nothing is rejected for being odd.** An unknown kind becomes `other`, numbers
that fail the game system's schema are kept verbatim in `extra` and the record
imports as text, a duplicate slug is suffixed. Each produces a warning the
importer is shown. A GM who uploads forty monsters and gets back thirty-nine
has no way to find out which one went missing.

**Nothing is half-imported.** Pack, records and audit row are one transaction.
Above 2 MiB the request reads only the *metadata* — cheap for all three shapes,
since a zip's central directory is indexed and the other two carry no manifest —
so the row is created with its real key before Celery gets the records, and a
re-import still matches. Failure lands on `status = 'failed'` with a reason
rather than deleting the row, because an admin who uploaded 30 MiB and came
back to nothing at all is #380 again.

**The uploaded file is not kept.** `source` holds the filename or URL, the
sha256 and the size. A staged large upload is deleted on success and on
failure. It waits under the audio volume — the one filesystem the backend and
worker share — in a directory whose name is deliberately not a UUID, which is
what keeps #406's orphan sweep from ever considering it.

**URL fetch follows its own redirects** so every hop is re-checked against the
private-address guard; letting httpx follow them would mean a public URL that
302s to 127.0.0.1 is fetched anyway, which is the whole of the SSRF. The
affirmation is checked before the request is made, so the endpoint cannot be
used as a blind fetcher.

**Search** keeps a campaign pack's version of a slug adjacent to the instance
pack's and puts the campaign's first, ranking the pair as a group — ordering by
the row's own name would split them the moment the GM titled theirs differently,
which is exactly the case the grouping exists for. Both are returned, each with
its source; hiding one would make "why can't I find it" unanswerable.

**Export defaults to withholding.** A record goes only if its own licence, or
the manifest of a native pack, says `redistributable`. Everything else is named
in the manifest with the reason, and every notice covering what is included
travels with it.

Endpoints per the spec: members search and read, admins own instance packs and
may disable/enable/remove *any* pack including a campaign's, GMs own their own.
Campaign routes answer 404 for an instance pack, not 403 — it genuinely is not
there, and 403 would confirm what it will not show.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs: content packs — the format, the endpoints, the changelog (#553 )

CI / Bot/backend version sync (pull_request) Successful in 50s

Details

CI / Backend lint (ruff) (pull_request) Successful in 53s

Details

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 1m22s

Details

CI / Docker image build (pull_request) Successful in 54s

Details

CI / Bot tests and audit (pull_request) Successful in 2m22s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 2m35s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 10m24s

Details

CI / Synthetic session harness (no GPU, no LLM) (pull_request) Successful in 15m43s

Details

48d13ceab3

Lane 1 of the content-packs feature is complete: `app/reference/` (the adapter
contract, the native reader in its three shapes, the sanitiser, the affirmation,
the import pipeline, search and export), the `content_packs` and
`reference_entries` tables behind migration `1c2d3e4f5a6b`, the member, admin
and GM endpoints, and 118 tests covering everything the spec's §11 names.

`docs/CONTENT-PACKS.md` is the reference a person writing a pack needs: what a
pack is, why Quest Board ships no content of its own, the affirmation text and
what is stored of it, a worked example of each of the three native shapes with
the CSV columns spelled out, what the importer does to the text on the way in,
how search decides what a caller may see, and — the section that will be read
in anger — which records export withholds and the exact reason it gives.

`docs/API.md` gains a Content Packs section in the existing format, placed
after Admin because it spans all three audiences rather than belonging to any
one of them.

The changelog entry is written for the people who feel the change: GMs get to
load the content they already have and look it up without leaving the app;
operators get a disable that takes anything on the instance out of search
immediately, keeps the rows, and lands in the audit log. It names the migration
and the two new dependencies, and says plainly that the screens follow with the
v4.4.0 interface work.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed

2026-09-08 06:45:48 +00:00

claude-bot merged commit 531c2a9674 into main

2026-09-08 07:02:15 +00:00

claude-bot deleted branch feat/553-content-packs-foundation

2026-09-08 07:02:16 +00:00

claude-bot referenced this pull request from a commit

2026-09-08 07:02:17 +00:00

Merge PR #557: feat(backend): content packs foundation: tables, import pipeline, search, export, endpoints (#553)

claude-bot referenced this pull request

2026-09-08 07:03:05 +00:00

[Feature] Rules and bestiary lookup at the table and in Prep #553