feat(backend): Open5e adapter and the starter-pack build scripts for content packs (#553) #564

Merged

claude-bot merged 4 commits from feat/553-open5e-adapter-starters into main 2026-09-08 08:34:51 +00:00

Conversation 0 Commits 4 Files changed 22 +3576 -26

claude-bot commented 2026-09-08 07:51:05 +00:00

Contributor

Copy link

Phase 1 of #553, lane 4 of the content-packs spec. Stacked on #562 (the Foundry fixes); merge that first and this PR's diff shrinks to its own four commits.

The Open5e reader. Saved v2 API pages, bare arrays, and open5e-*.zip bundles map to the shared kinds, with the SRD 5.1 / 5.2.1 CC-BY-4.0 attribution carried as the licence. Registered last in DETECT_ORDER; the detection cases for kilobyte-record arrays now answer open5e thanks to #562's closed native-array rule.

Two starter-pack builds, as scripts, not bundled content. scripts/content/build_5e_starter.py (Open5e's SRD documents, third-party documents dropped, CC-BY notice to Wizards) and build_pf2e_starter.py (the Foundry pf2e system's ORC-licensed records only, flavour text removed, records re-stamped with the ORC Notice and Paizo attribution). The pf2e build stages an ORC-only, flavour-free copy of each pack file and maps it through the real Foundry adapter, so there is one pf2e mapping in the tree. Both are pinned to an upstream commit and byte-identical on a re-run (SOURCE_DATE_EPOCH); the packs they produce are release assets fetched on request, never shipped in the image. Adapter warnings reach counts.warnings and a SOURCES.md section.

Also. The formats-endpoint test is lane 3's registry invariant; the starter script tests pin the rendered body (Statistics and Defences before Strikes, enrichers as prose) so a regression in the Foundry maps shows up here too.

Tests: 368 across the Open5e, starter-script, detection, Foundry, 5etools, orcbrew, import, admin, export, task, URL-fetch, search, game-systems and stat-schema suites. Fixtures are invented; the starter scripts are exercised against fixture trees, not live downloads.

🤖 Generated with Claude Code

Phase 1 of #553, lane 4 of the [content-packs spec](docs/design/content-packs-spec.md). Stacked on #562 (the Foundry fixes); merge that first and this PR's diff shrinks to its own four commits. **The Open5e reader.** Saved v2 API pages, bare arrays, and `open5e-*.zip` bundles map to the shared kinds, with the SRD 5.1 / 5.2.1 CC-BY-4.0 attribution carried as the licence. Registered last in `DETECT_ORDER`; the detection cases for kilobyte-record arrays now answer `open5e` thanks to #562's closed native-array rule. **Two starter-pack builds, as scripts, not bundled content.** `scripts/content/build_5e_starter.py` (Open5e's SRD documents, third-party documents dropped, CC-BY notice to Wizards) and `build_pf2e_starter.py` (the Foundry pf2e system's ORC-licensed records only, flavour text removed, records re-stamped with the ORC Notice and Paizo attribution). The pf2e build stages an ORC-only, flavour-free copy of each pack file and maps it through the real Foundry adapter, so there is one pf2e mapping in the tree. Both are pinned to an upstream commit and byte-identical on a re-run (`SOURCE_DATE_EPOCH`); the packs they produce are release assets fetched on request, never shipped in the image. Adapter warnings reach `counts.warnings` and a SOURCES.md section. **Also.** The formats-endpoint test is lane 3's registry invariant; the starter script tests pin the rendered body (Statistics and Defences before Strikes, enrichers as prose) so a regression in the Foundry maps shows up here too. Tests: 368 across the Open5e, starter-script, detection, Foundry, 5etools, orcbrew, import, admin, export, task, URL-fetch, search, game-systems and stat-schema suites. Fixtures are invented; the starter scripts are exercised against fixture trees, not live downloads. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 6 commits 2026-09-08 07:51:05 +00:00

fix(backend): a Foundry creature keeps its defences, its prose, and its format (#553) ... a14783abb3

Three gaps lane 4 found when it rewired the pf2e starter build through this
adapter.

**A creature lost half its stat block.** `body_sections` rendered strikes,
abilities and spells. Saves, skills, senses, languages and
immunities/weaknesses/resistances have no stat-schema field, so they survived
only inside `extra["system"]`, which nothing renders: a GM looking up an
imported creature saw what it hit with but not what it was immune to. Both
maps now emit two more sections before the strikes, in the order the book
prints them — Statistics (perception and senses, languages, skills) and
Defences (AC with the saves, HP with the IWR lists). AC and HP are repeated
from the envelope on purpose; a defence block listing Fort/Ref/Will but not
the AC beside them reads as though something went missing.

The 5e map cannot quite match it, and says so rather than pretending: that
system stores *proficiency*, not the computed bonus, so the rows name the save
and skill proficiencies instead of printing a number the file does not
contain. Deriving one from the challenge rating would be inventing data.

**Bare enrichers read as data.** `@Check[fortitude|dc:26|basic:true]` became
"fortitude, dc:26, basic:true". Most PF2e ability prose is made of these, so
every stat block read like a config file. The per-enricher renderer lane 4
wrote and then deleted when it adopted this adapter is recovered from
816a2e8 `scripts/content/_foundry_text.py` and folded into `_strip_enrichers`:
DC-and-statistic for @Check (including the pre-remaster `type:will` spelling),
"30-foot emanation" for @Template, formula-and-type for @Damage, the last
segment of a UUID path, and inline `[[/r …]]` rolls. Brackets are read by
counting rather than matched, because `@Damage[(4d6)[fire]]` nests.

**A bare Open5e array was still claimed as native.** 9deacde fixed this for
Foundry by declining on `_id`, but that is a rule about *someone else's*
format: an Open5e record has no `_id`, so a first record longer than the
512-byte sniff window — every real creature export — came straight back to the
native reader and imported as empty `other` records. Deciding by the absence
of other people's tells needs a new tell per format and fails silently each
time. The rule is now closed over our own shape: a truncated first element is
ours if `kind` is in the window, and `export_service._record_json` emits
`kind` first, which a test now asserts rather than trusts. The cost is a
hand-written array that buries `kind` past byte 512, refused loudly and fixed
with `format=questboard`; it is pinned as a test and documented in
CONTENT-PACKS.md beside the shape it constrains.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

test(backend): the malformed-pack task fixture leads with kind (#553) ... f7a7c94910

a14783a closed the bare-array rule: a truncated first element is native only
when `kind` sits inside the sniff window. `[{"name": "Truncated"` therefore
became nobody's format, and the task test that stages it asserted the parse
error ("not valid JSON") while getting the detection one ("could not tell what
format"). The fixture now leads with `kind`, as the rule and the docs ask of a
native array, so the failure under test is the parse again. The detection
failure is covered by the test above it.

Found by lane 4 of #553 while rebasing onto this branch; the hunk is theirs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(backend): Open5e adapter and starter-pack build scripts for content packs (#553) ... 5b791549c7

Lane 4 of the phase 1 content-packs work described in
docs/design/content-packs-spec.md: the Open5e adapter, and the two repo
scripts that build the redistributable starter packs.

app/reference/adapters/open5e.py reads the Open5e v2 shape in every form
people have it on disk — a saved API page, a bare array, a single record,
the project's per-resource data export, or a zip/folder of those — and
detects the resource from the record's shape. Creatures map onto the real
dnd5e stat schema (AC, HP, speed from speed_all, CR as a string with the
numeric CR as `level`, the six scores, size onto the schema enum, type as
creature_type); saves, skills, senses, languages, traits and actions become
Markdown.

Licensing is the whole of what makes this adapter interesting. Open5e has
no per-record licence field — a record's licence is its *document's* — so
DOCUMENT_LICENCES maps document keys to Licence objects: srd-2014 and
srd-2024 to CC-BY-4.0 with the SRD attribution, the known Kobold Press and
other third-party keys to OGL 1.0a, bfrd to CC-BY-4.0, and anything else to
None rather than a guess, so an unrecognised document imports as unlicensed
and is withheld from export.

scripts/content/build_5e_starter.py keeps only the two SRD documents and
writes a .qbpack whose manifest carries the CC-BY attribution;
build_pf2e_starter.py reads six Foundry pf2e packs, keeps only records whose
publication licence is ORC, and drops creature flavour text (publicNotes and
privateNotes) before mapping, because that is where the ORC License's
Reserved Material lives. Both write a SOURCES.md beside the pack, take
--pin/--limit/--dry-run, and produce byte-identical output on re-runs (sorted
records, fixed zip member timestamps, day-granularity created_at) so a
rebuild is a readable diff.

app/reference/adapters/base.py is the spec section 2 contract copied byte for
byte; lane 1 owns that file.

Every fixture is invented, per the fixtures policy in CLAUDE.md — invented
creatures, invented spells, invented rules. Only the document keys and field
names are real, because those are what the code filters on.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(backend): register the Open5e adapter and land the starters on the foundation (#553) ... b394732643

Rebased lane 4 onto the content-packs foundation and wired the two together.

- `adapters/base.py` is now the foundation's copy; the byte-for-byte stand-in
  this branch carried is gone.
- `open5e.py` exposes a module-level `adapter`, and `adapters/__init__.py`
  registers it on its own import line so lanes 2 and 3 can add theirs without
  touching the same one.
- Description text now goes through `app.reference.sanitize`, which is the one
  code path that tells HTML from Markdown: it cleans and converts anything
  tag-shaped with `to_markdown` and leaves a rule's Markdown `desc` alone,
  which markdownify would otherwise wreck. Slugs come from
  `app.reference.slugs.slugify` so an Open5e record and the same record from
  another format land on the same slug.
- Adapter-side warnings go in `meta.extra["warnings"]`: an unrecognised
  document (imported without a licence rather than guessing one), a pack that
  mixes licences, records with no usable name, and records whose shape matches
  no Open5e resource. None of them stop an import.

The starter scripts' manifests were checked against the reader that has to
accept them, `adapters/questboard.py`: `_meta_from_manifest` validates `format`
and `format_version`, copies `created_at`, `generator`, `counts` and `withheld`
verbatim, and ignores every other key — so the aggregate `withheld` shape and
the extra `source` block are both fine as they are. `source` is *dropped* on
import, though, so the upstream and the pin are now also folded into the pack
description, which survives onto the pack row. Two tests take a pack the script
just built, POST it to the admin import endpoint, and assert the counts, the
licence notice, the stats that passed validation, and — for pf2e — that the
flavour sentinels are absent all the way into the database.

`test_the_formats_endpoint_lists_the_registered_adapters` asserted the registry
held only `questboard`, which every adapter lane would have had to edit in turn.
It now checks the invariant instead: whatever is registered, in `DETECT_ORDER`,
questboard first.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

refactor(backend): the pf2e starter now maps through the Foundry adapter (#553) ... 96668d22c0

Rebased lane 4 onto lane 2 and deleted the mapping this script had been
carrying in the meantime.

`build_pf2e_starter.py` no longer knows how to read a Foundry document. The
mapping is `app.reference.adapters.foundry` — the same reader an admin's own
export goes through, so a published starter and a self-hosted import of the
same file now produce the same records. `_foundry_text.py`, the stopgap HTML
and enricher renderer, is gone with it.

What the script keeps is the release policy, which is not the adapter's
business: which six packs, ORC only, and the reserved-material rule. That rule
has to apply *before* the adapter sees a document — it reads
`details.publicNotes` as a creature's description and keeps the whole `system`
block in `extra` — so the kept documents are staged, cleaned, into a temporary
tree that mirrors `packs/<pack>/<file>.json`, and the adapter is pointed at
that. The mirrored layout is what keeps each record's `source_ref["path"]`
naming where it came from upstream. The ORC filter now reads the publication
block through `foundry_maps.pf2e.licence_block`, which knows all four places
the pf2e system has kept it.

Two consequences of handing the mapping over, both deliberate:

- A creature's summary would otherwise be the first heading of its strikes
  table, because the adapter summarises from the opening paragraph and the
  opening paragraph is exactly what the reserved-material rule deletes. The
  script fills that gap itself, since it made it.
- Saves, skills, senses, languages and immunities are no longer rendered into
  `body_md`; they survive in `extra["system"]`, where the adapter puts them.
  That is the adapter's shape and changing it belongs in `foundry_maps/pf2e`,
  not here.

Both build scripts now surface their adapter's `meta.extra["warnings"]`: as a
count in `counts.warnings` and as a section in SOURCES.md. For the 5e build the
section says plainly that the warnings are about the *source*, since that
adapter reads the whole export and the build then filters it.

Registry: questboard, foundry, open5e, one import and one `register` line each,
in DETECT_ORDER. `test_reference_detect.py` gains the Open5e boundary from this
side, including the case that does not work — a bare JSON array whose first
record overflows the 512-byte sniff window stays with the native reader,
because an Open5e record has no `_id` to give it away. It is pinned with the
two workarounds that do work, so closing it later is a decision rather than a
bug report.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs(changelog): the Open5e reader and the starter-pack builds (#553) ... caa5ac2bdf

Rebased onto lane 2's fix for the three gaps this lane found, and flipped
everything that was pinned as a gap into an assertion that it works.

The pf2e starter's creature bodies now carry the whole stat block — Statistics
and Defences before the strikes — and its enrichers read as prose again ("DC 26
basic Fortitude", "30-foot emanation", "4d6 fire"), so the starter-script tests
assert the real shape rather than what was missing from it. A bare Open5e array
is no longer claimed by the native reader; the two detect cases lane 2 left as
`!= "questboard"` pending this lane now read `== "open5e"`, and the "known gap"
test that pinned the old behaviour is gone because the behaviour is gone.

`headline()` stays. The adapter summarises from the opening paragraph, and with
the flavour paragraph removed the opening paragraph is `## Statistics` rather
than `## Strikes` — a better heading, still a heading.

One repair outside this lane: `test_a_malformed_pack_leaves_the_row_failed_with
_its_reason` staged `[{"name": "Truncated"`, which under the closed native-array
rule is nobody's format, so the row failed with "could not tell what format"
instead of the parse error the test is about — and the test above it already
covers that failure. The fixture now leads with `kind`, which is what the rule
and CONTENT-PACKS.md ask of a native array, and the test is about the parse
again. It fails on the fix branch without this change; it is not caused by
registering the Open5e adapter.

The CHANGELOG gains one bullet for the Open5e reader and the two build scripts:
what they build, that the packs are release assets rather than image content,
and the notices each carries. The foundation bullet loses "A reader for Open5e
follows", which it no longer does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

rbrooks force-pushed feat/553-open5e-adapter-starters from caa5ac2bdf to 59f4ffc44d 2026-09-08 08:14:27 +00:00 Compare

claude-bot scheduled this pull request to auto merge when all checks succeed 2026-09-08 08:14:34 +00:00

claude-bot merged commit d87c08c232 into main 2026-09-08 08:34:51 +00:00

claude-bot deleted branch feat/553-open5e-adapter-starters 2026-09-08 08:34:51 +00:00

claude-bot referenced this pull request from a commit 2026-09-08 08:34:53 +00:00

Merge PR #564: feat(backend): Open5e adapter and the starter-pack build scripts for content packs (#553)

claude-bot referenced this pull request 2026-09-08 08:38:18 +00:00

[Feature] Rules and bestiary lookup at the table and in Prep #553

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backend

Issue related to the backend

  

bot

Issue related to the bot

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

frontend

Issue related to the frontend

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

recording

Issue related to the recording system

  

security

Security hardening / vulnerability work

  

wontfix

This won't be fixed

No labels

backend

bot

bug

duplicate

enhancement

frontend

help wanted

invalid

question

recording

security

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/Quest-Board!564

No description provided.