Repository navigation
feat(ci): run plugin eval suites from dispatch and an opt-in PR label - #9835
Conversation
eval-browser-dispatch.yml now validates each suite with validate-cases.py before any model call, offers a deterministic-graders-only dispatch input, and runs on the `eval: run` label for a same-repository, non-draft pull request: each plugin whose skills, agents or evals the PR changes, with deterministic graders only, no granted tools, no scaffold, env scrub on and --max-cost-usd 5. summarize.mjs writes a with-versus-without table with PASS, FAIL or NOT COMPARABLE per case and a skill health line per case to the job summary. An empty credential secret ends the job with a notice. report-outcome adds the label when it opens a lane PR whose issue's done-when asks for a measured eval pass (eval-pass.mjs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DUS7XB8hUjcDnuSYLqJRzd
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
claude-review reviewed Incremental review of the change since ded9efc, in the four files listed. I found nothing to flag. What changed: What I read:
Closing issues (#9704, #9705): This increment doesn't touch their criteria. It tightens the allowlist and doesn't alter the path filter or the summary parser. Trusted paths: None of the four files matches a trusted-path glob. The change narrows what a pull request can pass through to the eval run, so it tightens the gate and does not widen it. Spend: None. No cost or billing change. |
|
claude-security-review reviewed Incremental review of the change since ded9efc: The change removes This narrows what a pull request can change, so it is not a trust-control widening. The new test covers the add, the edit and the unchanged case for both No security findings. critical-findings: 0 Not checked: I didn't read the |
The PR path's refusal was a denylist of load-time paths, decided from the live PR file list rather than the commit that runs. select-plugins.mjs now lets a selected plugin through only when every changed path is under evals/, or is markdown under skills/ or agents/ that keeps its non-descriptive frontmatter (hooks, allowed-tools, shell and the rest) and adds no ! shell preprocessing. The changed paths come from the checked-out head's own git diff against its merge base, so the refusal and the run read the same commit and the files API cap no longer applies. The concurrency group is keyed on the label, so another label event can no longer replace a pending eval re-run. The lane auto-label is removed (eval-pass.mjs, the report-outcome.mjs block and their tests, the docs sentence), and the label sender must be a listed human: neither issue asks for an unattended opt-in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
git diff --name-only quotes a path with unusual bytes and wraps it in double quotes, which the plugin path pattern did not match, so a quoted load-time path beside an eval change passed. Such a path is now refused, and changed paths are no longer trimmed before they are checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The frontmatter reader took only column-0 `key:` lines, so an indented block, a `? key` complex key, a flow mapping or frontmatter after a blank line hid an added hooks key from the comparison. Only a plain shape is now read (a `---` first line, a closing fence, unique column-0 keys with indented continuation lines); any other shape refuses the file. A YAML alias in a non-descriptive key is refused too, since it can pull in an anchor changed under a descriptive one. None of the 408 skill and agent files in the repository is refused on a body-only edit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t split frontmatter() split only on \r?\n, but a YAML reader also breaks lines on a bare \r and on U+0085, U+2028 and U+2029, so a hooks key after one was hidden from the comparison. A file holding any of them is now refused. None of the 408 skill and agent files in the repository holds one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
melodic-automation-lanes
Bot
added
the
needs-human
|
The pr-unblock lane stopped on this pull request and needs a person: the merge stages a change under .github/workflows/, which the lane does not push.
To resume, deal with the reason above, remove the gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=9835 -f activity=resolve-conflicts |
…f the public log On a pull request run the cases come from the PR, so the CLI's console output can carry model text. Write it to a file nothing uploads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # docs/conventions/loop-lane/CHANGELOG.md
melodic-automation-lanes
Bot
added
the
needs-human
|
The pr-unblock lane stopped on this pull request and needs a person: the merge stages a change under .github/workflows/, which the lane does not push.
To resume, deal with the reason above, remove the gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=9835 -f activity=resolve-conflicts |
kyle-sexton
removed
the
needs-human
…cy before a PR eval The pull request path denied Read of /home/runner/actions-runner, a guess at the runner's install directory. Read it from the Runner.Worker process above the step, deny it with both known layouts and the job's _temp and _actions directories, and stop the job when it is not found. A probe step then has one Claude Code run Read /proc/self/environ and the runner's .runner file, and stops the job unless the result's permission_denials records both reads as denied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
github-actions
Bot
added
the
needs-human
|
claude-security-review holds this pull request for a person (critical-security) at The claude-lane: claude-security-review |
…d unchecked plugins select-plugins.mjs let any path under a selected plugin's evals/ through, and a case's plugins: key can load a stub plugin there, or another plugin, whose hooks run outside the sandbox and can read the eval credential. - An evals/ path passes only as prompt.md, case.yaml, graders/*.md or samples/**/*.json. Scaffold scripts, fixtures (scaffold scripts run fixtures/ seed scripts) and stub plugin files are refused. - A changed prompt.md or case.yaml keeps env, context (scaffold_script), growthbook_overrides and every other run key; case.yaml's execution block may change only prompt, max_turns, timeout_seconds and allowed_tools. - Every case in a selected plugin loads only the plugin's root or an in-plugin directory with no changed path under it; anything outside the plugin is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p and Glob denials - The results artifact keeps its eval-browser-<run>-<attempt> name, the family .github/artifact-retention.json declares, so the retention check (lint-repo, test-bash) passes again. - The pull request probe now also asks for a Grep of /proc/self/environ and a Glob under /proc/self, requires a recorded denial for each, and fails if the result holds the credential anywhere. Read deny rules reach Glob and Grep only best-effort, so a call the policy misses stops the job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…! line A kept ! preprocessing line can start running when the text around it changes, such as an enclosing fence removed, with the line itself unchanged. A changed skill or agent file that already has one must keep its body; safe frontmatter keys may still change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A changed prompt.md or case.yaml must keep allowed_tools, top level and under execution, so a case changed in a pull request cannot ask for Bash or another gated tool; such a change runs by dispatch. Unchanged cases that list Bash get no Bash from the label either: a pull request run passes no --allow-tools, and a case cannot widen that grant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kyle-sexton
removed
do-not-merge
Post-merge mutation testingKilled 0, survived 1, timeout 0, no coverage 0, invalid 0, not run 7.
Surviving mutants
The run's summary has the full report. |
Closes #9704
Closes #9705
Summary
eval-browser-dispatch.ymlnow runs plugin eval suites two ways, under one job: by hand (dispatch on the default branch), and on theeval: runlabel for a same-repository, non-draft pull request. Every run checks its suite withvalidate-cases.pybefore any model call and keeps--max-cost-usd 5. Each run's JSON result goes throughsummarize.mjsinto the job summary. When the credential secret is empty, the job ends with a notice and runs nothing.Fix
pull_requestlabeledtrigger, gated on the label name, a same-repository head and non-draft. On that path,check-trusted-triggerchecks the trigger, and the label sender must be a listed human. It also adds adeterministic-onlydispatch input and acaseinput.targetmust now be aplugins/<name>directory. It keeps the kill switch, the read-only token and no write token. The PR path grants no tools, runs no scaffold, keeps env scrub on and skips the Chromium install. Onlyeval: runlabel events share a pull request's concurrency group, so another label cannot replace a pending re-run..github/scripts/plugin-eval/select-plugins.mjs: maps changed paths toplugins/<p>forskills/,agents/andevals/. The changed paths come from the checked-out head's owngit diffagainst its merge base, so the check and the run read the same commit. It is an allowlist: a selected plugin runs only when every changed path is underevals/, or is markdown underskills/oragents/that keeps its frontmatter outside the descriptive keys (hooks, allowed-tools, shell and the rest) and adds no!shell preprocessing. Frontmatter is read only in a plain shape (column-0 keys, indented continuations, closed fences); any other shape, a YAML alias in a non-descriptive key, or a path git quoted refuses the run..github/scripts/plugin-eval/deterministic-suite.mjs: copies the suite without cases that have anllmorbaselinegrader. The run then uses--eval-dir evals-deterministic..github/scripts/plugin-eval/summarize.mjs: applies the reference/ci.md parser order (partial, then skipped paid graders or errored runs, then a missing delta) and marks each such caseNOT COMPARABLE, with no score. Otherwise each case getsPASSorFAILagainst the run's threshold. Each case also gets a health line: whether the skill loaded in the with-arm and how many runs fired it. The last line isResult: <verdict>.lane-funding.jsonaddspull_requestto the eval lane's triggers. Loop-lane convention §7 documents the label (12.5.0).Verification
actionlint .github/workflows/eval-browser-dispatch.yml: exit 0.zizmor: no findings.node --testin.github/scripts/plugin-eval: 28 pass. These include the parser fixtures (scored, partial, skippedPaidGraders, errored; the last three print NOT COMPARABLE), the path filter (plugins/testing/evals/xselectsplugins/testingonly, anddocs/alone selects nothing), and the refusals: a script beside an eval change,bin/,commands/, non-markdown underskills/, a quoted path, frontmatter shapes a YAML reader parses differently (indented block,? key, flow mapping, blank line before the fence), an alias, and a SKILL.mdhooks:change read from a real git repository. None of the repository's 408 skill and agent files is refused on a body-only edit.node --testin.github/scripts/work-resolve-issue: 291 pass..github/actions/resolve-configcallers.test.mjs: 181 pass (lane-funding catalog).scripts/check-affected.sh --base origin/main --no-fetch: 9 passed, 4 skipped.Related
eval: runis declared underExtraLabelsfor claude-code-plugins in https://github.com/melodic-software/github-iac/pull/760. The org label grammar (colon then a space, never a bare colon) is why the name iseval: runand noteval:run.🤖 Generated with Claude Code