Skip to content

fix(guardrails): skip eval samples in reference-verify #10686

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kyle-sexton wants to merge 5 commits into from
Closed

fix(guardrails): skip eval samples in reference-verify #10686

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
cd4edb8
fix(guardrails): skip eval samples in reference-verify
kyle-sexton Oct 28, 2026
e264374
Merge branch 'main' into fix/guardrails-eval-samples-verify
kyle-sexton Oct 28, 2026
37104b5
docs(guardrails): cite #10694 in the eval-samples changelog fragment
kyle-sexton Oct 28, 2026
f6ca323
Merge remote-tracking branch 'origin/main' into fix/guardrails-eval-s…
kyle-sexton Oct 28, 2026
fa2ed4b
Merge main into fix/guardrails-eval-samples-verify
melodic-automation-lanes[bot] Oct 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changes/guardrails/fix-guardrails-eval-samples-verify-d4c705cd.md
View file
Open in desktop

@github-actions github-actions Bot Oct 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pr-verify verifier failed head e264374288fabd5042165079a059be8b1df926de. What it wrote, as data:

Summary: PR #10686 links #9936 with `Refs:`, so it has to deliver part of that issue. Issue #9936 asks for the developer-experience eval scenario sources to be committed under plugins/developer-experience/evals/sources/, scrubbed and reviewed for identifying details. This diff commits none of those sources. Instead it changes the guardrails `reference-verify` pre-push hook so it no longer checks files under any evals/**/samples/ folder. That is a prerequisite, not a part of the issue's scope. It also loosens a check the issue never asked to change. The PR body says 'Left out: nothing' and does not name the issue's remaining work. As a refs-scoped delivery it fails.
- PR body: for a `Refs:` link the body has to say which part of #9936 this delivers and what remains. It says 'Left out: nothing' and lists no remaining work: committing the sources, scrubbing the 11 files that contain a home path or email address, recording the identifying-details review, and the changelog check. Name what remains, or link the issue that actually covers this change.
- .changes/guardrails/fix-guardrails-eval-samples-verify-d4c705cd.md: the changelog fragment describes the hook change. Keep it with whichever issue ends up authorizing that change.

@kyle-sexton kyle-sexton Oct 28, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec for this hook change is now #10694, a sub-issue of #9936, linked with Closes #10694; the body names what #9936 still needs after this merges.

Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Fixed

- The `reference-verify` pre-push entry no longer verifies files under an `evals/` suite's `samples/` folder (#10694). Samples are recorded model output, so a CLI flag, skill reference or path inside one is data, and editing it would corrupt the eval. Prompts, graders and other eval files are still verified.
Loading