Repository navigation
Trade changelog-verification mechanics for the stakes behind them - #4056
Conversation
The verification section grew a procedure in #4051, and a procedure gets run literally and thinly. This trades some of it back for why the pass matters, stated where each reader can see it. The subagent never reads SKILL.md, so its template gets its own stakes: a claim that goes unchecked here goes out, and an entry that reads plausibly is not checked, because it was written from the same commits the verifier is about to read. SKILL.md's own line replaces "changelog mistakes are a recurring problem" with what a late correction costs. The workflow builds the release body with --notes-file at tag time (.github/workflows/release.yaml:381), so #4050's correction reached CHANGELOG.md and not the published v0.77.0 notes, which still read "Claude Code, Codex, OpenCode, and Gemini". Cut in exchange: step 2's enumeration of claim type to source, which was the overfit part, and a sentence from the clean-run gate. Scored against .claude/skills/release/evals/. A first pass also dropped "and confirming the old behavior there" from step 2; one of two runs then read the new file, concluded "schema 1 only", and reproduced the original error in its own suggested fix. Restored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N66ob4FWMHbGAGfQEcWHAp
worktrunk-bot
left a comment
There was a problem hiding this comment.
The stakes are the load-bearing part of this change, and as written they are not true of this repo's release path — the case cited as evidence is one where the correction did reach readers.
The mechanism half holds: the Create GitHub Release step in .github/workflows/release.yaml passes --notes-file to gh release create, so a later CHANGELOG.md edit is not picked up automatically. But the release body is editable, and #4050 edited it. That PR's own description says "The published GitHub release notes for v0.77.0 have been updated to match (they are generated from the tree at the tag, so they do not pick this up automatically)", and the live v0.77.0 body carries the corrections — I diffed its bullets against the ## 0.77.0 section on main and they are identical, branch_worktree_mismatch and the OpenCode 1.16 floor included. So "are not revised afterwards" — the sentence the verifier subagent actually reads, since it never sees SKILL.md — is the one claim #4050 falsified.
The stakes survive a true framing: the notification, feed, and mirror copies readers already have never change, and the release page changes only if someone edits it by hand. Suggestions inline for both lines. Stating it that way also keeps the post-tag remedy visible; as written, an agent that finds an error after the tag reads "not revised afterwards" and concludes there is nothing left to do, which is the opposite of what #4050 did.
On the cut: the clause dropped from step 2 covered claims whose truthmaker sits outside the commit — rendered output, docs, another component's own file. evals/README.md says of its own four cases "None of them has its truthmaker wholly outside the commit", so the battery cannot score a regression in that class by construction, and the eval table is silent on the cut rather than supporting it. The class is not hypothetical: #4050's OpenCode version-floor correction ("below 1.14.19 no longer loads it" against the documented 1.16 floor) is one, and that PR called it "wrong in a way users would act on". Worth deciding on its own rather than as part of "the overfit part" — a restore is suggested inline if you want it back.
The review disputed the stakes as written, and it was right. The published v0.77.0 release body carries all 29 corrected entries — it was hand-edited after the correction PR — so "not revised afterwards" is false. Verified by diffing the release body against the corrected CHANGELOG section: 29 entries, zero differences either way. The mechanism half holds: the workflow builds the body with --notes-file at tag time, so a later CHANGELOG edit isn't picked up. What that costs is a follow-up PR plus a hand-edit of the release page, with readers having acted in between. Both copies of the claim say that now, including the template's, which is the one the verifier reads. Also restores the clause dropped from step 2 for claims with no source in the commit. The eval set can't score that class — its README says none of its four cases has a truthmaker outside the commit — and the v0.77.0 errors included one: an OpenCode version floor whose source is OpenCode's own releases. The restored wording names that source, which the enumeration it replaces did not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N66ob4FWMHbGAGfQEcWHAp
#4051 fixed the changelog-verification section by adding procedure. A procedure gets run literally and thinly, so this trades some of it back for why the pass matters, stated where each reader can see it.
The subagent never reads SKILL.md. Stakes stated there reach the agent that spawns the verifier and nothing else, so the prompt template gets its own opener: the entries publish with the tag, and by the time anyone corrects a wrong line readers have acted on it — and reading an entry and finding it plausible is not a check, because it was written from the same commits the verifier is about to read.
SKILL.md's line now says what a late correction costs. It replaces "This is non-negotiable — changelog mistakes are a recurring problem". The workflow builds the release body with
--notes-fileat tag time (.github/workflows/release.yaml:381), so a laterCHANGELOG.mdedit isn't picked up: correcting a published entry takes a follow-up PR and a hand-edit of the release page.Cut in exchange: a sentence from the clean-run gate. Net +5 lines.
An earlier revision of this branch also cut step 2's guidance for claims whose source is outside the commit. The review caught that, and it's back — reworded, because the sentence I'd removed pointed at "the rendered output and the docs" and the docs would not have settled the OpenCode version floor that shipped wrong in v0.77.0. It now names the upstream project's own releases.
evals/README.mdstates that none of its cases has a truthmaker outside the commit, so the suite could not have scored that cut in either direction.Eval runs
Scored against
.claude/skills/release/evals/— four entries from the v0.77.0 section, three of them wrong — run in av0.77.0checkout with theSkilltool withheld, per that README.|| trueThe first draft had also dropped four words from step 2, "and confirming the old behavior there". Its failing run read the new file, found
json_output.rsdroppingWorktreeState::Detached, concluded "schema 1 only", and wrote a suggested fix that still claimedstateno longer reportsdetached— reproducing the original error while flagging the entry. That clause is restored, and the one miss in the three runs after it failed the same way.So the before-state case is 2/2 for #4051's wording and 2/3 for this one, which this design cannot separate: seven runs is far too few, and the case was already the weakest of the three when #4051 shipped it (0/2 before that PR's own fix, 1/2 after). What the eval supports is that relaxing the mechanics did not break the two solid cases, and that the four-word clause is load-bearing enough to keep. The argument for the stakes paragraphs is structural, not measured.
The guardrail entry was never called inaccurate in any of the seven runs. The runs predate the stakes correction below, which changes no instruction the cases exercise.
The stakes claim this PR opened with was wrong
The first version of this change claimed a post-tag correction reaches
CHANGELOG.mdand never the published release body, citing v0.77.0 as evidence. The review disputed it and was right: the published v0.77.0 body carries all 29 corrected entries, hand-edited after #4050. Diffed both ways, zero differences.I had grepped the published body for "Claude Code, Codex, OpenCode", matched the Pi entry's legitimate "markers … like Claude Code, Codex, OpenCode, and Gemini", and read it as the FAQ entry's uncorrected error — a plausible-looking string taken for the source, which is the failure this PR is about.
Fixed in 4800a75; only the mechanism that actually holds is claimed now.