[Review] Document every feature, and validate each doc against actual behaviour #442

New issue

Open

opened 2026-08-29 18:18:19 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-08-29 18:18:19 +00:00

Contributor

Part of the v4.6.0 release-readiness gate.

What

Document every feature and how to use it — and then prove each statement is true by exercising it. The validation half is not optional garnish; it is the half that gives the documentation any value.

Why the validation half matters more than the writing half

Documentation drifts silently, and this repo has the receipts:

get_vad_config's docstring said VAD "defaults to False so the transcribe path stays byte-identical" for a long time after it had defaulted to True — precisely backwards for anyone sizing the blast radius of a change.
check_transcript_covers_session's error told operators their capture clock was broken. It was measuring speech against wall clock and the capture clock was fine (#431).
apply_post_processing_audio_retention documented deleting audio inline as correct, reasoning "there is nothing left to reprocess". That reasoning cost a real session (#427).
docs/OPERATIONS.md instructed operators to run make backup-now before a risky upgrade, producing a dump nothing on the host could restore (#429).

In each case the prose was confident, specific, and wrong. A confidently wrong doc is worse than a missing one, because it stops the reader from checking.

Scope

Every user-facing feature: what it does, how to use it, what it will not do
Every admin setting: what it changes, what the default is, what happens if it is never set
Every operational procedure in docs/OPERATIONS.md
Every non-obvious invariant a future maintainer could break unknowingly

Validation method

For each documented claim, one of:

An automated test asserting the documented behaviour — strongest, and it keeps the doc honest afterwards.
A recorded manual exercise — the commands run and the output seen, pasted into the issue. Used for the #429 restore proof and for the v4.0.1 deploy verification.
Explicitly marked unverified, with the reason. Honest and useful; silent assumption is neither.

Where behaviour and documentation disagree, decide which is wrong — sometimes it is the code — and fix that one.

Acceptance criteria

Every feature has user-facing documentation covering purpose, usage and limits
Every admin setting documents its default and the consequence of leaving it unset
Every documented claim is validated by test, by recorded exercise, or explicitly marked unverified
Every operational procedure in docs/OPERATIONS.md has been run end to end, on a real host, and the output recorded
Discrepancies found are resolved in whichever layer is wrong, with the decision recorded
Docstrings that describe superseded behaviour are corrected — a stale docstring counts as a defect here, not a nitpick

#439 — a first-run wizard covers much the same ground for the operator; the settings inventory produced here is its input.

Part of the v4.6.0 release-readiness gate. ## What Document every feature and how to use it — and then **prove each statement is true** by exercising it. The validation half is not optional garnish; it is the half that gives the documentation any value. ## Why the validation half matters more than the writing half Documentation drifts silently, and this repo has the receipts: - `get_vad_config`'s docstring said VAD "defaults to False so the transcribe path stays byte-identical" for a long time **after it had defaulted to True** — precisely backwards for anyone sizing the blast radius of a change. - `check_transcript_covers_session`'s error told operators their capture clock was broken. It was measuring speech against wall clock and the capture clock was fine (#431). - `apply_post_processing_audio_retention` documented deleting audio inline as correct, reasoning "there is nothing left to reprocess". That reasoning cost a real session (#427). - `docs/OPERATIONS.md` instructed operators to run `make backup-now` before a risky upgrade, producing a dump nothing on the host could restore (#429). In each case the prose was confident, specific, and wrong. **A confidently wrong doc is worse than a missing one**, because it stops the reader from checking. ## Scope - Every user-facing feature: what it does, how to use it, what it will not do - Every admin setting: what it changes, what the default is, what happens if it is never set - Every operational procedure in `docs/OPERATIONS.md` - Every non-obvious invariant a future maintainer could break unknowingly ## Validation method For each documented claim, one of: 1. **An automated test** asserting the documented behaviour — strongest, and it keeps the doc honest afterwards. 2. **A recorded manual exercise** — the commands run and the output seen, pasted into the issue. Used for the #429 restore proof and for the v4.0.1 deploy verification. 3. **Explicitly marked unverified**, with the reason. Honest and useful; silent assumption is neither. Where behaviour and documentation disagree, decide which is wrong — sometimes it is the code — and fix that one. ## Acceptance criteria - [ ] Every feature has user-facing documentation covering purpose, usage and limits - [ ] Every admin setting documents its default and the consequence of leaving it unset - [ ] Every documented claim is validated by test, by recorded exercise, or explicitly marked unverified - [ ] Every operational procedure in `docs/OPERATIONS.md` has been run end to end, on a real host, and the output recorded - [ ] Discrepancies found are resolved in whichever layer is wrong, with the decision recorded - [ ] Docstrings that describe superseded behaviour are corrected — a stale docstring counts as a defect here, not a nitpick ## Related #439 — a first-run wizard covers much the same ground for the operator; the settings inventory produced here is its input.

claude-bot added this to the v4.6.0 - Release Readiness: Review, Simplification & Documentation milestone

2026-08-29 18:19:26 +00:00