feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663) - #681
Conversation
Two repo-local CI gates from the 2026-07-20 merged-PR quality audit, following the scripts/ + ci.yml precedent of the docs-only and cross-plugin-source-drift gates: self-testable, fail-closed, stale-guarded baselines that grandfather existing debt without red-lining it. Gate 1 (orphaned-fixture): every file under a skill's evals/fixtures/ must be consumed by a grader — an eval files[] entry or a test assertion. Gate 2 (CHANGELOG-parity): a versioned plugin must keep a CHANGELOG.md (static), and a PR that bumps a plugin's version must update its CHANGELOG.md in the same diff (--check-bump). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 3m 44s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
Overall this is a well-structured and thoughtful PR. The gate design is consistent with the repo's established patterns, the self-tests are comprehensive, and the baseline handling is carefully guarded against staleness. One correctness defect and two minor issues are worth addressing before merge. Correctness[Bug] Double stale-message in
When a grandfathered plugin (in the baseline) later ships a # path A — inside manifest loop
if [[ -f "$plugin_dir/CHANGELOG.md" ]]; then
if [[ -n "${grandfathered[$name]:-}" ]]; then
echo "STALE BASELINE: '$name' ..." # reported HERE
missing=$((missing + 1))
fi
continue # ← saw_debt["$name"] never set
fi
# path B — outer stale-scan loop
for name in "${!grandfathered[@]}"; do
if [[ -z "${saw_debt[$name]:-}" ]]; then
echo "STALE BASELINE: '$name' ..." # ALSO reported here (same entry)
missing=$((missing + 1))
fi
doneThe gate still exits 1 (fail-closed is preserved), and the test at line 75 only checks Fix: set if [[ -f "$plugin_dir/CHANGELOG.md" ]]; then
if [[ -n "${grandfathered[$name]:-}" ]]; then
echo "STALE BASELINE: '$name' in $BASELINE now has a CHANGELOG.md — remove it." >&2
missing=$((missing + 1))
saw_debt["$name"]=1 # ← prevents the second loop re-reporting this
fi
continue
fi[Minor]
base="${2:?usage: check-changelog-parity.sh --check-bump <base-ref>}"
Fix: replace with an explicit guard: if [[ -z "${2:-}" ]]; then
echo "usage: $(basename "$0") --check-bump <base-ref>" >&2
exit 2
fi
base="$2"[Minor] Fixture sort is not null-safe
if ((${#fixtures[@]} > 0)); then
mapfile -t fixtures < <(printf '%s\n' "${fixtures[@]}" | sort)
fiThis encodes fixture paths as newline-delimited input to Null-safe alternative using if ((${#fixtures[@]} > 0)); then
mapfile -t -d '' sorted < <(printf '%s\0' "${fixtures[@]}" | sort -z)
fixtures=("${sorted[@]}")
fiPositive observations
SummaryOne correctness bug (double stale message), one exit-code inconsistency, and one null-safety inconsistency. None block the gate from doing its job correctly — the fail-closed behaviour is preserved in all cases. The double-message bug is the most user-visible and the simplest to fix; the other two are cosmetic. The overall design, test coverage, and CI integration are sound. |
…p arg, null-safe sort Resolves the claude review-bot findings on #681: - check-changelog-parity.sh --check: mark a grandfathered plugin that gained a CHANGELOG.md as handled so the second stale-scan loop no longer double-reports it. - check-changelog-parity.sh --check-bump: explicit missing-arg guard exiting 2, consistent with every other usage-error path (${2:?} exited 1). - check-orphaned-fixtures.sh: null-safe fixture sort (sort -z / mapfile -d ''). - Tests: assert the stale message appears exactly once; cover --check-bump with no base ref -> exit 2. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 1m 24s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
All three findings from the prior review are correctly addressed in Fix 1 — Double stale-message: ✅ Correct
The test at Fix 2 — Exit-2 on missing bump arg: ✅ Correct
A new test case was added at Fix 3 — Null-safe sort: ✅ Correct
SummaryAll prior findings resolved cleanly. No new issues introduced. The PR is in good shape to merge. |
|
Claude finished @kyle-sexton's task in 2m 55s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
This review covers the full current diff at CorrectnessAll logic paths in both scripts are correct.
New-plugin edge case in Autonomy Security
CI wiring
Positive observations
SummaryAll prior findings resolved. No new issues found. The implementation is correct, the baselines are appropriately minimal and stale-guarded, the CI wiring is sound, and the test coverage is thorough. Ready to merge. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e03985e90c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…y prefix The autonomy entry grandfathered a whole directory (plugins/autonomy/skills/setup/evals/fixtures/), so any NEW orphaned fixture later added under it would match the prefix and be silently exempted — and because that directory already shadows many current orphans, the prefix never goes stale to flag the drift. The gate could pass wrongly (fail-open): a new ungraded fixture in this skill bypasses the check entirely. Replace the single directory prefix with an exact-path snapshot of the 173 currently-known orphans (otlp-demo + security-binding), same prefix-matching idiom as scripts/docs-only-paths.txt but each line pins one file. A new orphan added under an already-listed directory now matches no line and is red-lined; staleness still fires per-file when an entry's file is consumed or deleted. The knowledge youtube-digest entry was already an exact file and is unchanged. Verified at head e03985e: the enumerated set equals a fresh empty-baseline discover (174 = 173 autonomy + 1 knowledge); --check on the real tree passes; a probe fixture dropped under security-binding/ is now caught (exit 1) where the directory prefix would have exempted it; the script's own test suite stays green (7/7, including the stale-entry case). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 2m 38s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
This review covers the incremental change in Change SummaryThe only file changed in Correctness — Codex Finding Addressed Correctly ✅The Codex finding was: a directory-prefix entry grandfathers any future orphan added under that directory, not just the snapshot of current ones. The fix is mechanically correct:
The baseline comment at the top was updated to explain this explicitly:
Nit — Script Docstring Still Says "path-prefix lines"
The script header still describes entries as "path-prefix lines." With the baseline now using exact-file paths, this is slightly misleading — a reader might infer that directory-prefix entries are still the expected form. The baseline file's own comment has been updated correctly; the script header hasn't. This is cosmetic (the mechanism still works, and "path prefix" is technically accurate for an exact path), but a one-line clarification like "exact-path lines (each pins one file)" would keep the two in sync. No New IssuesNo correctness, security, or behavioral regressions were introduced. The self-tests and CI wiring are unchanged from the already-approved Summary
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e822423f03
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…re-changelog-parity-gates
…ixtures Merging origin/main brought in 7 new ungraded security-binding fixtures (escalation-*.json, from the WP runner-escalation lane) that no eval files[] entry or test consumes. The gate correctly red-lined them on the merge ref — it is catching NEW ungraded fixtures exactly as designed — so the exact-path baseline snapshot must be refreshed against the merged tree rather than the point where it was first cut. Add the 7 escalation-* paths to the snapshot (now 181 = 180 autonomy + 1 knowledge). This is debt-tracking, not a grader wiring: the escalation fixtures belong to the same ungraded security-binding class already owned by issue #662, whose cleanup scope now covers them. Verified on the merged tree (da0d5d4): the enumerated set equals a fresh empty-baseline discover (181, identical); --check passes; a probe fixture dropped under security-binding/ is still red-lined (exit 1); the script test suite stays green (7/7); changelog-parity --check passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 5m 8s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
One open finding: Codex P2 — baseline still uses prefix matchingThe Codex P2 finding from the Root cause — # Both functions use [[ "$path" == "$prefix"* ]] (prefix/glob)
# which matches valid.json AND valid.json.bak, valid.jsonl, etc.
is_grandfathered() {
local path="$1" prefix
for prefix in "${prefixes[@]}"; do
if [[ "$path" == "$prefix"* ]]; then # ← prefix matching, not exact
return 0
fi
done
return 1
}Fix — change both to exact equality: is_grandfathered() {
local path="$1" prefix
for prefix in "${prefixes[@]}"; do
if [[ "$path" == "$prefix" ]]; then # ← exact match
return 0
fi
done
return 1
}Same single-character change ( Update the test — the existing "grandfathered orphan" test at line 73 seeds a directory prefix ( # --- orphan grandfathered by an exact-path baseline entry -> passes ---------
repo="$(mk_repo $'plugins/p/skills/s/evals/fixtures/orphan.md\n')"
seed_skill "$repo" "plugins/p/skills/s" ''
printf 'x\n' >"$repo/plugins/p/skills/s/evals/fixtures/orphan.md"
if run_check "$repo" >/dev/null; then ok "grandfathered orphan passes --check"; else fail "grandfathered orphan wrongly failed"; fi
rm -rf "$repo"
# --- SYNTHETIC: same-name-prefix file NOT grandfathered by exact entry ------
repo="$(mk_repo $'plugins/p/skills/s/evals/fixtures/orphan.md\n')"
seed_skill "$repo" "plugins/p/skills/s" ''
printf 'x\n' >"$repo/plugins/p/skills/s/evals/fixtures/orphan.md" # satisfies entry
printf 'x\n' >"$repo/plugins/p/skills/s/evals/fixtures/orphan.md.bak" # must still be caught
out="$(cd "$repo" && bash scripts/check-orphaned-fixtures.sh --check 2>&1)"
rc=$?
if [[ $rc -ne 0 && "$out" == *"ORPHANED FIXTURE"*"orphan.md.bak"* ]]; then ok "same-prefix file is not grandfathered (exact match only)"; else fail "same-prefix file wrongly grandfathered: rc=$rc out='$out'"; fi
rm -rf "$repo"Also update the stale test's baseline entry from the directory prefix to an exact path, and update the inline comment at line 57 from "Active baseline prefixes:" to "Active baseline entries (exact paths):". Docstring — the script header at line 34 still says "path-prefix lines, same idiom as scripts/docs-only-paths.txt" (a nit flagged in the The fix is four mechanical edits: two Everything else is correctThe latest commit (
CI wiring — both lanes correctly included in The single remaining open item is the prefix-vs-exact matching bug described above. It's the last thing between this PR and a fully correct implementation. |
|
Baseline refreshed against a merged tree ( Merging Operational note on the snapshot idiom: the baseline is a merge-ref snapshot of known orphans, not a point-in-time cut. Because the exact-path lines intentionally do not grandfather whole directories, any ungraded fixture that lands on |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 57e8c69078
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ates Both gates could pass wrongly on inputs their own matching logic mishandled. 1. Orphaned-fixture baseline was consumed as a PREFIX, not an exact path. `[[ $path == $entry* ]]` meant a baselined `.../valid.json` also shadowed a new unconsumed `.../valid.json.bak` or `.../valid.jsonl` sibling, so an ungraded fixture could be added next to a grandfathered one and slip the gate. Match by full-string equality (`[[ $path == $entry ]]`) in both is_grandfathered and the staleness scan; rename prefix->entry throughout so the glob cannot be reintroduced. This also finishes closing the round-1 directory-grandfathering hole: an exact path shadows only itself. 2. CHANGELOG-parity --check-bump only checked `git diff --quiet` — that the changelog was touched at all. A whitespace/title/old-entry edit satisfied a version bump. Require the bumped head version's own `## [<version>]` entry to exist in the changelog at head (fixed-string grep, tolerant of a trailing date; a missing file counts as undocumented). Strictness is deliberate: a bump documented as `## 1.1.0` (no brackets) fails — enforcing the repo's Keep-a-Changelog `## [x.y.z]` heading format. Comments in both scripts and the orphaned-fixtures baseline header updated to describe exact-match / version-entry semantics (the prior "same idiom as docs-only-paths.txt" prefix framing no longer holds for orphaned fixtures). Tests: orphaned-fixtures gains a `.bak`/`.jsonl` sibling red-lines case and its two prefix-baseline cases become exact-path cases (8/8). changelog-parity gains a bump + unrelated-changelog-edit fails case; the pass case moves to the real `## [x.y.z]` format (12/12). Verified on the merged tree: both self-tests green; orphaned --check and changelog --check pass; the CI-identical --check-bump origin/main passes; a real `.bak` sibling of a baselined fixture now red-lines (exit 1) where the prefix match would have exempted it; shellcheck clean at --severity=style. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 1m 49s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
This review covers the latest commit ( Fix 1 — Exact-path matching in
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0b6f1b1303
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Follow-up hardening on the CHANGELOG-parity gate plus the normalization the gate now requires. 1. Normalize five plugins' CHANGELOG headings to the documented bracketed Keep-a-Changelog form (`## [x.y.z]`), preserving each file's existing date suffix: discovery, docs-hygiene, knowledge, playbooks, session-flow. These bump daily; unbracketed headings would false-fail --check-bump on their next bump as a CHANGELOG FORMAT error. Format-only — no version bumps, no content changes. 2. Split the --check-bump failure taxonomy. A bumped version documented with an unbracketed heading (`## <version>`) is now a distinct CHANGELOG FORMAT error that names the found heading and the expected bracketed form, rather than an UNDOCUMENTED BUMP. An author can fix format in place; conflating it with a missing release entry was misleading. 3. Close the pre-existing-entry fail-open. --check-bump passed whenever the bumped version's `## [<version>]` heading was merely present at head — even if it already existed in the base changelog, meaning the bump reused an old heading and shipped no new release note. Require the entry to be ADDED by the change set: present at head AND absent from the changelog at <ref>. A reused pre-existing heading is now reported as PRE-EXISTING CHANGELOG ENTRY. 4. Cite issue #688 in the orphaned-fixtures baseline comment for variation-matrix-backlog.json, replacing the un-owned "needs triage" note now that the fixture has a needs-triage grade-or-demote issue. Tests: check-changelog-parity gains a FORMAT-split case, a newly-added-entry pass case, and a pre-existing-entry fail case (15/15). Verified on the merged tree (branch + origin/main): both self-tests green, --check and the CI-identical --check-bump origin/main green; shellcheck clean at --severity=style. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 3m 55s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
This review covers the incremental change in Fix 1 — Pre-existing-entry fail-open: ✅ CorrectThe last open Codex P2 finding (classified "VALID defer" by kyle-sexton with the carve-out note that it's a fail-open) is resolved in this commit. Mechanism: if [[ -f "$changelog" ]] && grep -Fq "## [$head_version]" "$changelog"; then
if git show "$base:$changelog" 2>/dev/null | grep -Fq "## [$head_version]"; then
echo "PRE-EXISTING CHANGELOG ENTRY: ..."
preexisting=$((preexisting + 1))
fi
continue # always — avoids double-counting
fiThe New file case handled correctly: When a CHANGELOG is brand-new in this PR (absent at Fix 2 — Format-taxonomy split: ✅ CorrectA bumped version documented as esc="${head_version//./\\.}"
if found="$([[ -f "$changelog" ]] && grep -m1 -E "^##[[:space:]]+${esc}([[:space:]]|\$)" "$changelog")"; thenThe dot-escaping is correct ( Fix 3 — CHANGELOG normalization (five plugins): ✅ Format-onlydiscovery, docs-hygiene, knowledge, playbooks, session-flow — all changes are mechanical bracket additions only ( Test suite: ✅ Three new cases, 15/15
The pre-existing test (case 8) correctly constructs a base commit where the plugin version is No new issuesAll prior approved findings remain fixed. No correctness, behavioral, or security regressions were introduced. CI wiring and both baselines are unchanged. Summary
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c1c3830c17
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ding Codex P2 on #681: the fixed-string version match also accepted the bumped version appearing in prose or an indented/fenced example, satisfying (or falsely pre-existing) the release entry without a real heading. Both the head and base checks now match an anchored Markdown heading; self-test case added proving a prose/indented mention fails as UNDOCUMENTED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FM1RfM3jHkgenpdbMv4o64
|
Claude finished @kyle-sexton's task in 2m 7s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
This review covers the incremental change in Fix — Line-start anchor on the release-entry match: ✅ CorrectThe Codex finding was: Both call-sites are now anchored identically. esc="${head_version//./\\.}"
heading_re="^## \[${esc}\]"
if [[ -f "$changelog" ]] && grep -Eq "$heading_re" "$changelog"; thenAnd the base check at if git show "$base:$changelog" 2>/dev/null | grep -Eq "$heading_re"; thenRegex construction is correct. For
A line-start match correctly rejects:
A trailing
New test case: ✅ Correct and specific
The assertion checks for Test suite is now 16/16, matching kyle-sexton's verification note. The test count in the PR description was 15/15 for the prior round; the new case correctly adds one. No new issuesThe change is tightly scoped to the two Summary
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8c028dcebf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ng check Two Codex P2s on #681, one per gate: - orphaned-fixture: grep -w treated "." as a word boundary, so a referenced valid.json also consumed a new unconsumed valid.json.bak sibling. Basename matches now require non-filename-character neighbors around an ERE-escaped basename. - changelog-parity: the heading match interpolated the version into an ERE with only dots escaped, so SemVer build metadata (1.0.1+build.1) broke the pattern and false-failed a documented bump. The heading check is now a fixed-string line-anchored awk match (no regex at all); the FORMAT-split probe escapes all ERE metacharacters. Self-tests: suffix-sibling orphan case + build-metadata round-trip case added; orphan suite 9/9, parity suite 17/17. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FM1RfM3jHkgenpdbMv4o64
|
Claude finished @kyle-sexton's task in 5m 11s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
This review covers the latest commit ( Fix 1 — Suffix-sibling consumption (orphaned-fixture gate): ✅ CorrectThe Codex finding:
esc_base="$(printf '%s' "$base" | sed -E 's/[][\\.|$(){}?+*^]/\\&/g')"
base_re="(^|[^A-Za-z0-9._-])${esc_base}([^A-Za-z0-9._-]|$)"The character class Test coverage —
Both assertions in one test, proving the fix closes the hole without false-positiving the legitimate fixture. Suite is now 9/9. Fix 2 — SemVer build-metadata in heading regex (changelog-parity gate): ✅ CorrectThe Codex finding: the
heading="## [${head_version}]"
has_heading() { awk -v h="$heading" 'index($0, h) == 1 { found=1; exit } END { exit !found }'; }
The FORMAT branch's ERE probe at line 169–170 now escapes ALL ERE metacharacters ( Test coverage —
Suite is now 17/17. No new issuesThe changes in
CI wiring, baseline files, and the SummaryBoth Codex P2 findings from the |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 91571eb1d7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…re-changelog-parity-gates
|
Claude finished @kyle-sexton's task in 4m 20s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
This review covers the current HEAD ( Finding 1 — Path substring match is a real fail-open ✅ VALID
if grep -qF "$rel" "$evals_json"; then
return 0
fi
Why the existing test does not catch this: The suffix-sibling test at The fix is to apply the same character-class-bounded ERE approach to # Path match: require boundaries so evals/fixtures/valid.json.bak in files[]
# does not consume the separate, unconsumed evals/fixtures/valid.json.
esc_rel="$(printf '%s' "$rel" | sed -E 's/[][\\.|$(){}?+*^]/\\&/g')"
if grep -qE "(^|[^A-Za-z0-9._-])${esc_rel}([^A-Za-z0-9._-]|$)" "$evals_json"; then
return 0
fiAfter The test should add a case seeding Finding 2 —
|
…d headings Two Codex P2s on #681, both fail-open in the gates' matching: 1. check-orphaned-fixtures.sh: the files[] PATH check used an unbounded fixed-string search of the whole evals.json, so an eval referencing evals/fixtures/valid.json.bak marked a NEW unconsumed evals/fixtures/valid.json sibling consumed (shorter $rel is a substring of the longer value). Compare $rel against each JSON string value EXACTLY with jq; the bounded-basename check remains the fallback. (Complements the earlier basename-boundary fix, which left this path branch open.) 2. check-changelog-parity.sh: the awk release-heading predicate matched `## [<version>]` at column zero without tracking fenced-code state, so a bump adding only a ``` fenced example containing the heading satisfied the gate — an undocumented release. The predicate now toggles fence state on ``` / ~~~ lines and matches only outside fences; the overclaiming comment is corrected. Self-tests: orphaned adds a files[]-substring case (10/10); changelog adds a fenced-block case (18/18). shellcheck clean at --severity=style and against .shellcheckrc; orphaned --check + set-equality (181, unchanged — the stricter path match reclassified no real consumption), changelog --check, and --check-bump origin/main all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FM1RfM3jHkgenpdbMv4o64
|
Claude finished @kyle-sexton's task in 2m 48s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
This review covers the latest commit ( Fix 1 —
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: edf355b917
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…k fence closing check-orphaned-fixtures.sh: evals.json consumption now reads only files[] values (any nesting) — a fixture named in a prompt or metadata string no longer counts as consumed; exact-value and bounded-basename matches run against those values only. check-changelog-parity.sh: fence tracking records the opening delimiter char + length and closes only on a matching delimiter per CommonMark, so a ~~~ content line inside a backtick fence no longer re-opens heading matching. One regression test each. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 3m 9s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
This review covers the latest commit ( Fix 1 —
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0c1bcc3098
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…appease typos gate A close now requires the same delimiter char, a run at least as long as the opener, a whitespace-only suffix, and at-most-three-space indentation — a ```not-a-close content line or a four-space-indented would-be closer no longer clears fence state, so a release heading inside that fence cannot satisfy the parity gate. Opening backtick fences reject an info string containing a backtick. Regression test added; test-message wording adjusted for the spell-check gate (hyphen-split false positive on a prefixed word). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 4m 29s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
This review covers the latest commit ( Fix — Closing-fence suffix and indentation validation: ✅ CorrectThe Codex finding: the prior fence-close condition
if (match($0, /^ {0,3}`+/) || match($0, /^ {0,3}~+/)) {
seg = substr($0, RSTART, RLENGTH)
sub(/^ +/, "", seg)
mchar = substr(seg, 1, 1); mlen = length(seg)
rest = substr($0, RSTART + RLENGTH)
if (mlen >= 3) {
if (!infence) {
if (!(mchar == "`" && rest ~ /`/)) { infence = 1; fchar = mchar; flen = mlen; next }
} else if (mchar == fchar && mlen >= flen && rest ~ /^[ \t]*$/) {
infence = 0; next
}
}
}Indentation constraint ( Closing-suffix constraint ( Delimiter-type isolation ( Backtick info-string constraint (opening: The four constraints together match CommonMark §4.5 fenced code blocks exactly. Test coverage: ✅ 20/20
[1.0.0] |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 600f1d281e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…HTML-comment headings check-orphaned-fixtures.sh: a bounded-basename match in a test file counts only inside the fixture's own skill; a test elsewhere in the plugin must name the fixture by its plugin-relative path — same-named fixtures in sibling skills are no longer conflated. check-changelog-parity.sh: has_heading tracks multi-line HTML-comment state alongside fences — a release heading inside <!-- --> is not rendered Markdown and neither satisfies nor pre-exists the entry; comment markers inside fences stay content, fence markers inside comments stay suppressed. Regression tests for all three shapes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 4m 20s —— View job PR Review — feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663)
This review covers the latest commit ( Fix 1 — Skill-scoped test consumption (
|
| Case | What it proves |
|---|---|
| Sibling-skill basename conflation (lines 88–101) | skills/a test naming shared.md → skills/b shared.md is still caught; skills/a shared.md is not |
| Cross-skill consumption by plugin-relative path (lines 103–111) | A plugin-root test referencing skills/b/evals/fixtures/by-plugin-test.md correctly marks it consumed |
Suite is now 13/13.
Fix 2 — HTML-comment heading exclusion (check-changelog-parity.sh): ✅ Correct
The Codex finding: has_heading tracked fenced-code state but not HTML comment state, so ## [version] appearing at column zero inside a `` block would satisfy the release-entry check.
scripts/check-changelog-parity.sh lines 161–205: the has_heading awk now tracks multi-line HTML comment state (inhtml) in parallel with fence state (infence):
- Block 1 (first, highest priority): if
!infence && inhtml, scan the line for-->to close the comment and callnext— all lines inside an HTML comment are skipped for heading matching and fence detection. - Block 2: fence detection (unchanged from prior round).
- Block 3:
!infence && index($0, h) == 1— heading match, skipped inside fences (and HTML comments via block 1'snext). - Block 4:
if (!infence) { HTML comment tracking }— scan for `` on unfenced, non-comment lines; setsinhtml=1on an unclosed opener.
Invariant: infence=1 and inhtml=1 cannot be simultaneously true. Block 4 is guarded by !infence, so inhtml can only become 1 when outside a fence. Block 1 is guarded by !infence && inhtml, but since inhtml can't be 1 while inside a fence, block 1's fence guard is a safety belt, not a reachability condition.
Interaction properties verified:
- Fence markers inside HTML comments: block 1 fires first with
next, fence detection (block 2) never runs → fence markers inside comments are suppressed ✓ - HTML comment markers inside fenced code: block 4 is guarded
!infence, so ``: handled inline in block 4'swhileloop — the heading check never ran (block 3 requires column 1, which a post-`
[1.0.0]
Asserts `UNDOCUMENTED BUMP` and NOT `PRE-EXISTING` — both the head check and base check correctly reject the comment-enclosed heading. Suite is now **21/21**.
---
### No new issues introduced
All prior approved findings remain correctly resolved at `efb9717`:
| Finding | Status |
|---|---|
| Exact-path baseline matching (no prefix leak) | ✓ |
| Stale-baseline deduplication (single message per entry) | ✓ |
| Exit-2 on missing `--check-bump` arg | ✓ |
| Null-safe sort via `sort -z` | ✓ |
| Suffix-sibling basename boundary (`.bak` not consumed by `valid.json` reference) | ✓ |
| Exact `files[]` path match via jq | ✓ |
| PRE-EXISTING / FORMAT / UNDOCUMENTED failure taxonomy | ✓ |
| `^## [<version>]` line-start anchor via `awk index()==1` | ✓ |
| SemVer build-metadata via `awk index()` fixed-string match | ✓ |
| Fenced-code tracking (column-zero heading inside fence rejected) | ✓ |
| CommonMark fence-type isolation (mismatched delimiter doesn't close) | ✓ |
| `files[]`-only consumption scope (prose mention doesn't count) | ✓ |
| CommonMark fence closing (delimiter char + length + whitespace-suffix + ≤3-space indent) | ✓ |
| Test-basename scoped to owning skill | ✓ *(this commit)* |
| HTML-comment heading exclusion | ✓ *(this commit)* |
CI wiring and both baselines are unchanged.
---
### Summary
Both Codex P2 findings from the `600f1d2` round are correctly resolved in `efb9717`. The orphaned-fixture gate now uses a two-loop design that confines basename matches to the fixture's owning skill while still allowing cross-skill consumption via unambiguous plugin-relative paths. The changelog-parity gate's `has_heading` predicate now tracks HTML comment state alongside fence state, correctly rejecting headings that appear at column zero inside `` blocks. Both fixes have targeted synthetic test cases proving the failure modes are closed. No new issues introduced. The PR is ready to merge.
Resolves a CHANGELOG heading conflict: main bracketed all session-flow CHANGELOG.md version headings to satisfy the new CHANGELOG-parity CI gate (#681); this branch's new 0.10.3 entry predated that reformat. Applied the bracketed `## [0.10.3]` form to the new entry so it matches the rest of the file.
…s, zero orphans (#708) ## Summary Fix-direction (a) from the issue, per the #634 graded-fixture idiom: every fixture under `plugins/autonomy/skills/setup/evals/fixtures/security-binding/` is now graded. - **One table-driven runner** (`check-security-binding.fixtures.test.mjs`) + **one co-located expectations manifest**: each entry pins the checker invocation (`--probe-evidence-root` at the fixtures dir; per-fixture `--egress-hosts`/`--evidence` where needed) and the expected outcome — exit code plus defect-naming stderr substrings. 109 fixtures: 14 pass-expected (12 valid bindings + 2 evidence-input pairings), 95 reject-expected. **Zero quarantined — no name-vs-behavior mismatches surfaced.** - **Self-policing both directions**: a new top-level fixture without a manifest (or quarantine) entry fails; a manifest ref whose file vanished fails; the 67 `probe-transcripts/` suite inputs are enumerated and reconciled against disk both ways. - **Baseline drained**: all 178 security-binding lines leave `scripts/orphaned-fixtures-baseline.txt`; the orphaned-fixture gate (#681) passes with the set consumed, exactly as its stale-guard demands. - Thin `.test.sh` wrapper joins `plugins/**` CI test discovery. Suite: **394/394 checks pass.** `validate-plugin-contracts.mjs`, orphaned-fixtures `--check`, changelog-parity `--check`/`--check-bump` all green. Autonomy plugin bumped 0.7.3 with CHANGELOG entry. Note for reviewers: the convention-level decision about the golden-fixture idiom repo-wide stays with #664 (needs-human); this PR instantiates the already-precedented #634 shape for the one suite #662 names, which the issue's own fix-direction (a) authorizes. ## Related - #681 (the orphaned-fixture gate whose baseline this drains) - #664 (repo-wide golden-fixture convention decision — untouched) Closes #662 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary The `--check-bump` gate's PRE-EXISTING-ENTRY arm compared **every** versioned plugin's manifest against the base ref. When `main` advanced any plugin's version while a PR was open, the PR's stale branch red-lined on a plugin it never touched — forcing an unnecessary merge-from-main on every base advance. Observed twice on #681 (work-items 0.14.2 → 0.14.4; #681 never touched work-items). ## Fix Scope the bump loop to plugins whose **manifest** changed in the branch's own diff: - `git diff --name-only base...HEAD` — three-dot isolates `diff(merge-base(base,HEAD), HEAD)`, the commits unique to this branch. A version `main` advanced after the branch forked is excluded whether CI checks out the PR head or the auto-merge commit. - The filter keys on the manifest path `plugins/<name>/.claude-plugin/plugin.json`, **tighter than the issue's "plugin roots" wording**. A version bump is definitionally a change to that file, so manifest-scoping is precisely "plugins whose version this change set could have changed" — and, unlike plugin-root scoping, a cosmetic touch elsewhere under a plugin dir cannot pull a main-only advance back into scope (the same treadmill, just triggered by a stray edit). Plugins the branch never touched are left out of the check entirely; plugins the branch did touch are checked exactly as before. ## Verification Full self-test suite (extended with three branch-staleness cases: main-only advance not flagged, the discriminating cosmetic-touch/manifest-scoping case, and a counterpart proving a branch-bumped plugin is still checked): ``` $ bash scripts/check-changelog-parity.test.sh ... ok: untouched plugin advanced only on the base ref is not flagged (branch staleness scoped out) ok: cosmetic touch under a plugin dir does not pull a main-only advance into scope (manifest-scoped) ok: a plugin the branch bumped is still checked while the main-only advance is scoped out ... PASS=24 FAIL=0 ``` Live check against the real repo: ``` $ scripts/check-changelog-parity.sh --check-bump origin/main Every plugin whose version changed vs origin/main has a '## [<version>]' CHANGELOG.md entry. ``` `shellcheck` clean on both files. Closes #693 ## Related - #693 — this fix - #681 — the PR where the re-merge treadmill was first observed --------- Co-authored-by: Claude <noreply@anthropic.com>
… evals/fixtures (#727) ## Summary `plugins/knowledge/skills/youtube-digest/evals/fixtures/variation-matrix-backlog.json` sat under a skill's `evals/fixtures/` but no eval case referenced it (no `files[]` entry in the sibling `evals.json`) and no test asserted on it. It was grandfathered in `scripts/orphaned-fixtures-baseline.txt` — the last un-owned entry there (the autonomy `otlp-demo` fixtures are tracked by #662). ## Fix **Decision: demote** (not grade). The file is a manual smoke-test tracking backlog — candidate videos across footage variations (code screencast / slide talk / talking-head / mixed) carrying `status` fields (`smoke-pass`), acquisition notes, and blocked-caption records. It is reference/tracking data, not an input→expected-output graded fixture (the genuine graded fixture in this skill is `driver-video-goldens.json`, wired via `files[]` in evals 2 and 4). `SKILL.md` itself labeled it "backlog only". - Moved `evals/fixtures/variation-matrix-backlog.json` → `reference/variation-matrix-backlog.json` (out of the graded-fixture scope, into the skill's `reference/` per the issue's steer). - Repointed the two prose references at the new path: `SKILL.md` (now describes it as a tracking backlog, not an eval fixture) and vendor `TUNING.md`. - Removed its grandfather line (and its comment block) from `scripts/orphaned-fixtures-baseline.txt`. - Bumped `plugins/knowledge` `0.8.0 → 0.8.1` (patch: relocation + docs, no behavior change) with a top-inserted `CHANGELOG.md` entry. ## Verification Ran on the rebased branch tip: ``` $ scripts/check-orphaned-fixtures.sh --check No orphaned eval fixtures (every file under **/evals/fixtures/ is consumed by a grader or grandfathered). # exit 0 $ scripts/check-orphaned-fixtures.sh discover | grep variation-matrix # (no output — file no longer under evals/fixtures/) $ scripts/check-changelog-parity.sh --check Every versioned plugin has a CHANGELOG.md (or a stale-guarded baseline entry). # exit 0 $ scripts/check-changelog-parity.sh --check-bump origin/main Every plugin whose version changed vs origin/main has a '## [<version>]' CHANGELOG.md entry. # exit 0 $ node scripts/validate-plugin-contracts.mjs Plugin contracts validated: 33 setup skills and 1816 plugin files checked. # exit 0 $ jq empty reference/variation-matrix-backlog.json # relocated file still valid JSON # exit 0 ``` Closes #688 ## Related - #688 — this issue (grade-or-demote the orphaned fixture) - #663 — orphaned-fixture CI gate that surfaced it - #681 — PR that introduced the gate + baseline Co-authored-by: Claude <noreply@anthropic.com>
Summary
Two repo-local CI gates from the 2026-07-20 merged-PR quality audit, both wired into
.github/workflows/ci.ymlas new required lanes. Each follows the established repo-local gate precedent (#593/#609/#622/#628): a self-testable script, a dedicated CI step, fail-closed behavior, and clear diagnostics — with a stale-guarded baseline that grandfathers pre-existing debt without red-lining fixtures/plugins a member issue already owns. Placement is repo-local only; promotion toci-workflowsis explicitly deferred ("if it stabilizes") and out of scope.No plugin version bump: the CI scripts and
ci.ymlcarry no plugin version to bump. The onlyCHANGELOG.mdedits in this PR are a format-only normalization of five pre-existing unbracketed changelogs (see Gate 2) — no version fields change. This repo has no repo-root version artifact.Fix
Gate 1 — orphaned-fixture (
scripts/check-orphaned-fixtures.sh, laneorphaned-fixture-gate). Repo-wide static scan of every file under a skill's**/evals/fixtures/. A fixture is consumed when its skill-relative path appears in the siblingevals/evals.json(an evalfiles[]entry), or its basename appears — bounded by non-filename characters, so a referencedvalid.jsondoes not also mark avalid.json.baksibling consumed — in thatevals.jsonor any*.test.*file in the plugin. Consumption matching errs toward not-blocking a legitimate fixture. Scope isevals/fixtures/specifically — unit-test fixture dirs (tests/fixtures,scripts/fixtures) are excluded because those are often generated or loaded by directory, not named, and a name-matcher cannot honestly grade them.--checkfails on an un-grandfathered orphan and on a stale baseline entry (one shadowing no orphan), so the baseline cannot outlive its debt.Existing debt grandfathered in
scripts/orphaned-fixtures-baseline.txt— exact fixture paths, matched by full-string equality (not prefix), so the baseline is a snapshot of today's known orphans: a new orphan under an already-listed directory, or a<name>.json.bak/<name>.jsonlsibling of a listed file, matches no line and is red-lined rather than silently grandfathered. Every entry cites an owning issue:plugins/autonomy/skills/setup/evals/fixtures/corpus (security-binding golden suite +otlp-demosample corpus), enumerated file-by-file — tracked by autonomy: 16 security-binding fixtures (~1,200 lines) are orphaned — no eval references them, no test grades them #662; autonomy is WP-lane-owned.plugins/knowledge/skills/youtube-digest/evals/fixtures/variation-matrix-backlog.json— a data file named only fromSKILL.md/vendor prose, consumed by no grader — tracked by knowledge youtube-digest: variation-matrix-backlog.json eval fixture is orphaned — grade or demote #688 (grade-or-demote).Gate 2 — CHANGELOG-parity (
scripts/check-changelog-parity.sh, lanechangelog-parity-gate). Covers both gaps the audit named:--check(static, every event): aplugins/<name>/.claude-plugin/plugin.jsoncarrying aversionmust shipplugins/<name>/CHANGELOG.md. Catches the cited violator —autonomyshipped 5 minor bumps with no CHANGELOG.md.--check-bump <base>(PR-only, mirrors thesync-*.sh --check-bumpgates): a manifest version change must add a## [<version>]release heading for the new version — present at head, absent at<base>. The heading is matched as a fixed string anchored to line start, so the version appearing in prose or a fenced example never satisfies (or falsely pre-exists) the entry, and SemVer build metadata (1.0.1+build.1) never leaks into a regex. Three distinct failures, never conflated:UNDOCUMENTED BUMP(no entry — an unrelated whitespace/title/old-release edit cannot satisfy the gate),CHANGELOG FORMAT(the version is present but written unbracketed,## <version>), andPRE-EXISTING CHANGELOG ENTRY(the heading already existed at<base>, so the bump shipped no fresh note). Applies to every plugin, grandfathered or not.Enforcing the bracketed
## [<version>]Keep-a-Changelog heading surfaced five pre-existing changelogs still on the unbracketed## <version>form (discovery,docs-hygiene,knowledge,playbooks,session-flow); this PR normalizes them to the documented convention (format-only, headings only — no version or entry-content changes).Reconstructing
autonomy's history is autonomy-plugin (WP-lane) work, so it is grandfathered by name inscripts/changelog-parity-baseline.txt(stale-guarded:--checkfails if it later gains a CHANGELOG.md or drops its version). The baseline never relaxes--check-bump.Both lanes are registered in the
ci-statusneeds:aggregate (the single source of truth for required lanes).Verification
All output below is real, run in the worktree against
origin/main.Self-tests (run unconditionally in CI so a broken detector cannot mask a regression):
Both gates pass on current HEAD (existing debt grandfathered):
Live synthetic catch — Gate 1 (orphaned fixture, no baseline):
Live synthetic catch — Gate 2 (missing changelog, no baseline):
Static analysis:
shellcheck --rcfile .shellcheckrcclean on all four new scripts;actionlint .github/workflows/ci.ymlclean; new.shfiles committed mode 100755, all files LF.Closes #663
Related
knowledgeyoutube-digest orphan (grandfathered by Gate 1's baseline).ci.ymllane + fail-closed) this follows structurally.🤖 Generated with a Claude Code implementation subagent (issue #663)