Skip to content

Trade changelog-verification mechanics for the stakes behind them - #4056

Merged
max-sixty merged 2 commits into
mainfrom
diagnose-release-notes-accuracy
Sep 9, 2026
Merged

max-sixty merged 2 commits into
mainfrom
diagnose-release-notes-accuracy

Conversation

@max-sixty

@max-sixty max-sixty commented Sep 9, 2026 •

Copy link
Copy Markdown
Owner

#4051 fixed the changelog-verification section by adding procedure. A procedure gets run literally and thinly, so this trades some of it back for why the pass matters, stated where each reader can see it.

The subagent never reads SKILL.md. Stakes stated there reach the agent that spawns the verifier and nothing else, so the prompt template gets its own opener: the entries publish with the tag, and by the time anyone corrects a wrong line readers have acted on it — and reading an entry and finding it plausible is not a check, because it was written from the same commits the verifier is about to read.

SKILL.md's line now says what a late correction costs. It replaces "This is non-negotiable — changelog mistakes are a recurring problem". The workflow builds the release body with --notes-file at tag time (.github/workflows/release.yaml:381), so a later CHANGELOG.md edit isn't picked up: correcting a published entry takes a follow-up PR and a hand-edit of the release page.

Cut in exchange: a sentence from the clean-run gate. Net +5 lines.

An earlier revision of this branch also cut step 2's guidance for claims whose source is outside the commit. The review caught that, and it's back — reworded, because the sentence I'd removed pointed at "the rendered output and the docs" and the docs would not have settled the OpenCode version floor that shipped wrong in v0.77.0. It now names the upstream project's own releases. evals/README.md states that none of its cases has a truthmaker outside the commit, so the suite could not have scored that cut in either direction.

Eval runs

Scored against .claude/skills/release/evals/ — four entries from the v0.77.0 section, three of them wrong — run in a v0.77.0 checkout with the Skill tool withheld, per that README.

before-state hooks || true five agents vs three rows guardrail
#4051's wording 2/2 2/2 2/2 held
first draft here 1/2 2/2 2/2 held
this branch 2/3 3/3 3/3 held

The first draft had also dropped four words from step 2, "and confirming the old behavior there". Its failing run read the new file, found json_output.rs dropping WorktreeState::Detached, concluded "schema 1 only", and wrote a suggested fix that still claimed state no longer reports detached — reproducing the original error while flagging the entry. That clause is restored, and the one miss in the three runs after it failed the same way.

So the before-state case is 2/2 for #4051's wording and 2/3 for this one, which this design cannot separate: seven runs is far too few, and the case was already the weakest of the three when #4051 shipped it (0/2 before that PR's own fix, 1/2 after). What the eval supports is that relaxing the mechanics did not break the two solid cases, and that the four-word clause is load-bearing enough to keep. The argument for the stakes paragraphs is structural, not measured.

The guardrail entry was never called inaccurate in any of the seven runs. The runs predate the stakes correction below, which changes no instruction the cases exercise.

The stakes claim this PR opened with was wrong

The first version of this change claimed a post-tag correction reaches CHANGELOG.md and never the published release body, citing v0.77.0 as evidence. The review disputed it and was right: the published v0.77.0 body carries all 29 corrected entries, hand-edited after #4050. Diffed both ways, zero differences.

I had grepped the published body for "Claude Code, Codex, OpenCode", matched the Pi entry's legitimate "markers … like Claude Code, Codex, OpenCode, and Gemini", and read it as the FAQ entry's uncorrected error — a plausible-looking string taken for the source, which is the failure this PR is about.

Fixed in 4800a75; only the mechanism that actually holds is claimed now.

This was written by Claude Code on behalf of max-sixty

The verification section grew a procedure in #4051, and a procedure gets
run literally and thinly. This trades some of it back for why the pass
matters, stated where each reader can see it.

The subagent never reads SKILL.md, so its template gets its own stakes: a
claim that goes unchecked here goes out, and an entry that reads plausibly
is not checked, because it was written from the same commits the verifier
is about to read.

SKILL.md's own line replaces "changelog mistakes are a recurring problem"
with what a late correction costs. The workflow builds the release body
with --notes-file at tag time (.github/workflows/release.yaml:381), so
#4050's correction reached CHANGELOG.md and not the published v0.77.0
notes, which still read "Claude Code, Codex, OpenCode, and Gemini".

Cut in exchange: step 2's enumeration of claim type to source, which was
the overfit part, and a sentence from the clean-run gate.

Scored against .claude/skills/release/evals/. A first pass also dropped
"and confirming the old behavior there" from step 2; one of two runs then
read the new file, concluded "schema 1 only", and reproduced the original
error in its own suggested fix. Restored.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N66ob4FWMHbGAGfQEcWHAp

@worktrunk-bot worktrunk-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stakes are the load-bearing part of this change, and as written they are not true of this repo's release path — the case cited as evidence is one where the correction did reach readers.

The mechanism half holds: the Create GitHub Release step in .github/workflows/release.yaml passes --notes-file to gh release create, so a later CHANGELOG.md edit is not picked up automatically. But the release body is editable, and #4050 edited it. That PR's own description says "The published GitHub release notes for v0.77.0 have been updated to match (they are generated from the tree at the tag, so they do not pick this up automatically)", and the live v0.77.0 body carries the corrections — I diffed its bullets against the ## 0.77.0 section on main and they are identical, branch_worktree_mismatch and the OpenCode 1.16 floor included. So "are not revised afterwards" — the sentence the verifier subagent actually reads, since it never sees SKILL.md — is the one claim #4050 falsified.

The stakes survive a true framing: the notification, feed, and mirror copies readers already have never change, and the release page changes only if someone edits it by hand. Suggestions inline for both lines. Stating it that way also keeps the post-tag remedy visible; as written, an agent that finds an error after the tag reads "not revised afterwards" and concludes there is nothing left to do, which is the opposite of what #4050 did.

On the cut: the clause dropped from step 2 covered claims whose truthmaker sits outside the commit — rendered output, docs, another component's own file. evals/README.md says of its own four cases "None of them has its truthmaker wholly outside the commit", so the battery cannot score a regression in that class by construction, and the eval table is silent on the cut rather than supporting it. The class is not hypothetical: #4050's OpenCode version-floor correction ("below 1.14.19 no longer loads it" against the documented 1.16 floor) is one, and that PR called it "wrong in a way users would act on". Worth deciding on its own rather than as part of "the overfit part" — a restore is suggested inline if you want it back.

Comment thread .claude/skills/release/SKILL.md Outdated
Comment thread .claude/skills/release/SKILL.md Outdated
Comment thread .claude/skills/release/SKILL.md Outdated
The review disputed the stakes as written, and it was right. The
published v0.77.0 release body carries all 29 corrected entries — it
was hand-edited after the correction PR — so "not revised afterwards"
is false. Verified by diffing the release body against the corrected
CHANGELOG section: 29 entries, zero differences either way.

The mechanism half holds: the workflow builds the body with
--notes-file at tag time, so a later CHANGELOG edit isn't picked up.
What that costs is a follow-up PR plus a hand-edit of the release page,
with readers having acted in between. Both copies of the claim say that
now, including the template's, which is the one the verifier reads.

Also restores the clause dropped from step 2 for claims with no source
in the commit. The eval set can't score that class — its README says
none of its four cases has a truthmaker outside the commit — and the
v0.77.0 errors included one: an OpenCode version floor whose source is
OpenCode's own releases. The restored wording names that source, which
the enumeration it replaces did not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N66ob4FWMHbGAGfQEcWHAp
@max-sixty
max-sixty merged commit ebf8624 into main Sep 9, 2026
49 checks passed
@max-sixty
max-sixty deleted the diagnose-release-notes-accuracy branch September 9, 2026 04:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants