[max-sixty]

The v0.77.0 changelog verification pass reported 16 accurate, 6 needing revision, 0 incorrect. Eleven factual errors reached users anyway and were corrected in #4050. This changes the verification section so the pass covers the text that ships.

Two mechanisms produced the eleven, both visible in the pass's own report.

Five of the errors were sentences the verifier itself wrote. The template asks it to "quote a shorter rewrite", and the section's gate read "do not finalize until the subagent confirms every entry" — a gate you pass once. So the rewrites were applied and shipped with nothing having read them. The gate is now a clean run over CHANGELOG.md as it stands, and the text says why: a rewrite the verifier suggests is a new draft with no more evidence behind it than one you wrote yourself.

Four were non-headline clauses inside entries stamped accurate. The FAQ entry is the clearest: the verifier's evidence line says "three paths" and its verdict says accurate, under an entry claiming five agents. The template now verifies claim by claim rather than entry by entry, and its report format asks for the source read per claim, so a contradiction like that one lands on the page. The others were claims the diff cannot settle — a prior behavior, an untouched file, rendered output, a documented version floor — against a template whose one strong instruction ("read the actual diff") installed the diff as the sole source.

The template also said "[paste drafted entries]" while the gate below it said to re-run over CHANGELOG.md as it stands — the same drift channel one layer down, and how the failing pass came to verify a draft the working tree had already moved past. It now names the extraction one-liner the section already carries, so the rule is a mechanism rather than an instruction, and the gate drops the clause the template enforces. Found in review.

The six-step list that duplicated the prompt template below it is gone, so the section's line count is flat.

Measurement: A/B over the shipped 29-entry section, two runs per arm

Each arm ran the verification template as a subagent prompt over the whole v0.77.0 section, with the earlier per-group notes offered as a map — the load the failing pass had. Handed only the four scenario entries, every wording found everything, so that design measured nothing; attention spread across 29 entries is the variable.

scenario	old	first draft	shipped
before-state claim, diff looks confirming	0/2	0/2	1/2
falsifying fact in an added comment	1/2	2/2	1/2
non-headline clause, settled in the diff	0/2	1/2	2/2
accurate entry stays clean (guardrail)	2/2	2/2	1/2, one soft scope flag

The first draft of step 2 failed the before-state case in both runs, so it was sharpened and retested rather than shipped on the strength of reading well. The cases are kept in evals/README.md.

No case has its truthmaker wholly outside the commit, and that is hard to arrange here: this repo's commits routinely restate the surrounding facts in added comments and doc lines, so the falsifying text usually arrives in the diff. The rows score how much of the diff the verifier reads, not whether it goes looking beyond one. The second row was labelled "truthmaker outside the diff" until review pointed out that 29a611b7c states the fact in a comment it adds beside the fix.

The arms were run with the section pasted into the prompt, so the measurement covers the verification procedure rather than how the entries arrive. Now that the template resolves its own input, evals/README.md runs the arm in a checkout at v0.77.0, where the top section is that same 29 entries — so the template runs unedited. Review caught that the first version of this change left the harness scoring ## Unreleased and reporting a clean pass.

The clean-run gate is not in that table. It changes the loop rather than the subagent prompt, so two arms of the prompt cannot separate it; it is argued structurally.

This was written by Claude Code on behalf of max-sixty

🤖 Generated with Claude Code

[worktrunk-bot]

The claim-by-claim rewrite and the clean-run gate both hold up, and I verified all four eval cases against the commits they cite — 981a207 has no Detached variant before the PR adds it, every v0.76.0 marker hook ends || true, and the FAQ table 5a552b2 added has three rows with Gemini in none of them. Two points below, plus one on where the cases live.

evals/README.md is reachable only by listing the skill directory — nothing in SKILL.md names it. The reader who needs it is whoever next edits the verification section, and they are reading SKILL.md. Without a pointer the cases stop getting re-run, and the section drifts back the way this PR is correcting.

[worktrunk-bot]

evals/README.md says to run the template as a subagent prompt over the v0.77.0 section, but as of this push the template resolves its own input — "the top section of CHANGELOG.md as it stands on disk". Run verbatim, an arm now scores whatever release is in progress (today, the two ## Unreleased entries), reports nothing about the three known-wrong cases, and reads as a clean pass rather than a broken harness. That only surfaces when someone re-runs the cases to score a change to the template, which is the moment the section is drifting back. Suggestion inline.