[melodic-automation-lanes]

Closes #10694

Summary

Opened by the work-resolve-issue lane. The run's own summary:

The reference-verify pre-push entry no longer hands files in a samples/ folder below an evals/ folder (root or nested, directly or deeper) to cli-flag-verify, skill-reference-verify and stale-path-verify. Those files are recorded model output, so a flag, skill reference or path in them is data, and editing them to pass corrupts the eval. Everything else stays verified, including an eval suite's prompts, graders and fixtures, and any samples/ folder outside evals/.

Not verified

• plugins/guardrails/git-hooks/reference-verify.test.sh was not executed: this lane's Bash runs only the named wrappers, and the review subagent's attempt was denied. Left to the PR's CI
• Pushing ＃9936's eval-sources branch through the hook was not done: the lane cannot reach that branch or run the hook. The added test cases model its paths (evals/sources/labels/T1/samples/run1.md). Left to a person or the ＃9936 push
• tidy skipped passes. reference-verify.sh rows were dropped because a script wired as a git hook is on tidy's GLOBAL HARD list. Pass 1 applied one simplification to eval_case in the test file. Pass 2 found no zero-information comments. Pass 3 (docs-hygiene:write-for-humans) was skipped for the turn b
• typos check skipped by check-affected (typos not installed); CI-only checks listed by check-affected are left to the PR's CI

Fix

• plugins/guardrails/git-hooks/reference-verify.sh: verified_type, the one filter that picks which changed files reach the three verifiers, now returns false for evals/samples/, evals//samples/, /evals/samples/ and /evals//samples/; the header comment and the function comment say so
• plugins/guardrails/git-hooks/reference-verify.test.sh: new eval_case helper. Stay-quiet cases: an unknown flag in plugins/p/evals/c/samples/answer.md, evals/sources/labels/T1/samples/run1.md and evals/samples/x.md does not block. Must-fire cases: the same flag blocks in plugins/p/evals/c/graders/che
• plugins/guardrails/README.md: the reference-verify entry row says which files the entry skips and that the rest of an eval suite stays checked
• .changes/guardrails/fix-10694-reference-verify-eval-samples-3f9a1c7e.md: new patch changelog fragment for guardrails, named by hand because new-changelog-fragment refuses a detached HEAD

Verification

The runner ran each check from the base copy against 0180ecd:

• scripts/check-changelog-fragments.sh --check: passed

The run reports:

• check-changelog-fragments (after the edit, after tidy, after the review fixes): All 2 changelog fragment(s) under .changes/ are valid
• check-affected (run three times, same points): 12 passed, 6 skipped, no failures; the skips are validate-plugins sub-checks that run code from the checked tree, ruff (no .py), typos (not installed) and changed-skills (no skill file)
• fresh-context review subagent over the whole diff: no correctness bugs. It confirmed that the case globs match exactly samples/ below evals/ and reject myevals/, evals-x/, xsamples/ and docs/samples/, that cli-flag-verify scans inline code spans in .md so the stay-quiet cases fail without the fix, a
• The lane run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40732936415

Left out: the items under Not verified.

Related

#10694

[melodic-automation-lanes]

Unattended merge: native auto-merge leaves this pull request unarmed at head 0180ecd9baf7652b2050823ba818a6b1c23e6ca9. The lanes clear this reason on their own; the next scheduled run judges it again.

pr-merge's gate refuses its verifier verdict (verify:missing)

[github-actions]

claude-security-review reviewed 0180ecd9baf7652b2050823ba818a6b1c23e6ca9: 0 finding(s), 0 as inline comments.

No security issues found.

critical-findings: 0

Scope: the diff has 4 files. plugins/guardrails/git-hooks/reference-verify.sh gets a new case arm in verified_type. The other three are its test, a README row and a changelog fragment. No changed file matches a trusted path, and none is a plugin manifest, workflow, token or kill-switch control, or pr-pipeline.yaml. There are no manifest <field> items to report.

What I checked:

• Widened skip. The change makes the local pre-push reference-verify entry skip .md, .sh, .bash, .ps1 and .psm1 files under evals/**/samples/. That entry runs cli-flag-verify, skill-reference-verify and stale-path-verify. These check that references are accurate. They are not secret scanners, authorization gates or merge holds, and pre-commit-content (secrets and machine-specific paths) does not use verified_type. A file in a skipped folder therefore still gets the secret and path checks at commit.
• Not a merge gate. A pre-push hook runs on the contributor's machine and can be bypassed with --no-verify. It is not a CI merge hold.
• Glob precision. The patterns match only a path segment named exactly evals with samples/ below it. A look-alike directory such as myevals/samples/x.md is not skipped, and the new tests cover that case. The * in a shell case pattern also matches /, so nested folders are skipped, which is intended.
• Injection. The change is a literal case pattern. No new input is interpolated into a command. In the tests, the $p arguments are fixed literals passed as quoted arguments.
• Instruction-surface deletions. The diff narrows what a hook scans. The skipped material is recorded eval output, and no protected-class rule is removed. I did not read docs/conventions/instruction-exception-register/README.md. I judged by consequence instead: a missed stale reference in an eval sample is a quality issue, not a security one.

Spend: none. The change adds no billed resource and no new CI runs.

[github-actions]

claude-review reviewed 0180ecd9baf7652b2050823ba818a6b1c23e6ca9: 0 finding(s), 0 as inline comments.

No findings. I read the diff and the surrounding code in reference-verify.sh and reference-verify.test.sh. I did not run the test script, because this lane has no shell.

• Filter: verified_type in plugins/guardrails/git-hooks/reference-verify.sh is the only gate before the three verifiers (reference-verify.sh:160). In a bash case pattern * matches /, so the four globs cover samples/ at any depth below evals/, at the repository root or nested. myevals/samples/, docs/samples/ and evals/…/graders/ are not matched and stay verified.
• Closing issue guardrails: reference-verify skips recorded eval samples (evals/**/samples/) #10694, rule: the rule is met. The change narrows only the file filter. Prompts, graders and fixtures stay verified, and so does a samples/ folder outside evals/.
• Closing issue guardrails: reference-verify skips recorded eval samples (evals/**/samples/) #10694, tests: the stay-quiet and must-fire cases in the test file cover every path the issue lists, plus evals/samples/x.md and myevals/samples/x.md. The stay-quiet cases would fail without the filter, since cli-flag-verify reads inline code spans in .md. Two criteria are not met in this review. I did not run the full test script. The issue's last item, pushing chore(developer-experience): commit the hand-rolled eval scenario sources #9936's eval-sources branch through the hook, is not done and the PR body lists it as not verified, so a person or the chore(developer-experience): commit the hand-rolled eval scenario sources #9936 push has to confirm it.
• Docs: the README row, the header comments and the changelog fragment (bump: patch) agree with the code.
• Spend: none of the change spends API credits or can cause a charge.

[melodic-automation-lanes]

The unattended merge path armed native auto-merge at head 0180ecd9baf7652b2050823ba818a6b1c23e6ca9.

[github-actions]

Post-merge mutation testing

• Range: 9f89f97..f41aa45
• Merged pull request: #10698
• Outcome: reported

Killed 1, survived 0, timeout 0, no coverage 0, invalid 0, not run 0.
Covered-code score 100%; mutation score 100%.

File	Killed	Survived	Timeout	No coverage	Invalid	Not run
plugins/guardrails/git-hooks/reference-verify.sh	1	0	0	0	0	0

The run's summary has the full report.