Skip to content

guardrails: reference-verify skips recorded eval samples (evals/**/samples/) #10694

Closed

Description

@kyle-sexton

Slice of #9936.

Problem

The guardrails pre-push reference-verify entry blocked the push of #9936's eval scenario sources with 28 findings, every one dotnet build --nologo (not in 'dotnet build --help'), and every one in a labeled judge sample under plugins/developer-experience/evals/sources/labels/*/samples/. Those files are recorded model output. A flag, skill reference or path inside them is data, not a claim the repository makes (docs/conventions/untrusted-content/README.md), and eval cases keep recorded output as recorded (.claude/rules/eval-case-transcripts.md). Editing them to pass the check would corrupt the eval, so the check is a false positive on these files.

This is a hook-precision fix (docs/conventions/hook-precision/README.md): it narrows which files a verifier reads. It does not change what the verifiers enforce on authored text, and it is not a security-control change.

Rule

  • Skip only files in a samples/ folder below an evals/ folder (at the repository root or nested, directly or deeper), in the one filter that picks which changed files go to the three verifiers (cli-flag-verify, skill-reference-verify, stale-path-verify).
  • Prompts, graders, fixtures and harness files of an eval suite stay verified.
  • A samples/ folder outside evals/ stays verified.

Done when

  • A stay-quiet test that fails before the fix: an unknown flag in plugins/p/evals/c/samples/answer.md and in evals/sources/labels/T1/samples/run1.md does not block the push.
  • Must-fire tests pass before and after the fix: the same unknown flag still blocks in plugins/p/evals/c/graders/check.md, plugins/p/evals/c/prompt.md and docs/samples/x.md.
  • plugins/guardrails/git-hooks/reference-verify.test.sh passes in full.
  • The guardrails README entry row says which files the entry skips.
  • Pushing chore(developer-experience): commit the hand-rolled eval scenario sources #9936's eval-sources branch through the hook reports no findings in the samples/ files.

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 28, 2026
  2. github-actions commented on Oct 28, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: scoped
    Basis: automated intake triage; the lanes App applied this class.

    Size: Small
    Complexity: Low
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Understood: the guardrails pre-push reference-verify hook gives false positives on recorded eval samples under evals/**/samples/. The proposed fix narrows the one file filter feeding the three verifiers, adds stay-quiet and must-fire tests, and updates the README row. The scope and done-when items are clear. The final item (pushing the #9936 branch) depends on that branch and may need a separate run. No duplicates found among the open issues.

    Automated triage, applied by the lanes App.

  3. added
    priority: mediumReal value, no hard deadline; normal backlog flow.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 28, 2026
  4. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Oct 28, 2026
  5. added a commit that references this issue on Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Type

    Fields

    No fields configured for Bug.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions