Repository navigation
Description
Slice of #9936.
Problem
The guardrails pre-push reference-verify entry blocked the push of #9936's eval scenario sources with 28 findings, every one dotnet build --nologo (not in 'dotnet build --help'), and every one in a labeled judge sample under plugins/developer-experience/evals/sources/labels/*/samples/. Those files are recorded model output. A flag, skill reference or path inside them is data, not a claim the repository makes (docs/conventions/untrusted-content/README.md), and eval cases keep recorded output as recorded (.claude/rules/eval-case-transcripts.md). Editing them to pass the check would corrupt the eval, so the check is a false positive on these files.
This is a hook-precision fix (docs/conventions/hook-precision/README.md): it narrows which files a verifier reads. It does not change what the verifiers enforce on authored text, and it is not a security-control change.
Rule
- Skip only files in a
samples/folder below anevals/folder (at the repository root or nested, directly or deeper), in the one filter that picks which changed files go to the three verifiers (cli-flag-verify,skill-reference-verify,stale-path-verify). - Prompts, graders, fixtures and harness files of an eval suite stay verified.
- A
samples/folder outsideevals/stays verified.
Done when
- A stay-quiet test that fails before the fix: an unknown flag in
plugins/p/evals/c/samples/answer.mdand inevals/sources/labels/T1/samples/run1.mddoes not block the push. - Must-fire tests pass before and after the fix: the same unknown flag still blocks in
plugins/p/evals/c/graders/check.md,plugins/p/evals/c/prompt.mdanddocs/samples/x.md. -
plugins/guardrails/git-hooks/reference-verify.test.shpasses in full. - The guardrails README entry row says which files the entry skips.
- Pushing chore(developer-experience): commit the hand-rolled eval scenario sources #9936's eval-sources branch through the hook reports no findings in the
samples/files.
Activity
github-actions commentedon Oct 28, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood: the guardrails pre-push reference-verify hook gives false positives on recorded eval samples under evals/**/samples/. The proposed fix narrows the one file filter feeding the three verifiers, adds stay-quiet and must-fire tests, and updates the README row. The scope and done-when items are clear. The final item (pushing the #9936 branch) depends on that branch and may need a separate run. No duplicates found among the open issues.
Automated triage, applied by the lanes App.
docs(guardrails): cite #10694 in the eval-samples changelog fragment