[kyle-sexton]

Closes #11624

Summary

An emitted invocation-probe case runs in an empty workspace, so the agent searched outside it with Glob, Grep and Read. run-validity.py failed the run on those denied calls, so no trigger measurement could gate VALID. Probe runs now pass --disallow-tools Read Glob Grep, which removes the search tools. A case's allowed_tools cannot do this: it only adds tools.

Fix

• The README measure-invocation.sh emit-plugin-eval writes beside the cases gives the full run line: --ablation none --keep-temp --disallow-tools Read Glob Grep.
• reference/invocation-probes.md "emit-plugin-eval results" states the flag and why. It points at claude plugin eval --help as the source, with an as-of date and recheck trigger, because the plugin-evals page documents grants only.
• /playbooks:skill-authoring rewrite-a-plugin.md step 5 runs the emitted probes with the flag.
• The emitted allowed_tools is unchanged, so a run that forgets the flag gets the old denial pattern rather than the worse one allowed_tools: [Skill] gave.

Verification

• Host smoke (Claude Code 2.1.316, Opus 5.5): the 19 emitted /ai-slop:audit probe cases, with-arm, --runs 1. run-validity.py verdict VALID, 0 denied tool calls in 19 traces. Before the fix, 6 of 8 runs had denials, and 16 of 19 with allowed_tools: [Skill].
• Per-case outcomes matched the earlier smoke except one positive that now fires (18/19 vs 17/19). That is no trigger-rate change beyond one-run noise, which meets the done-when.
• measure-invocation.test.sh: all passed, including a new check that the emitted README carries the flag; it fails on main.
• scripts/check-affected.sh: 14 passed.
• Left out: a 3-run measurement on this skill alone. The Paid judge-calibration and suite run for the evals plugin after #5879's sample fixes #9155 probe measurement now running uses this flag on 156 cases.

Related

• Unblocks the before/after trigger measurements for Paid judge-calibration and suite run for the evals plugin after #5879's sample fixes #9155 and ai-slop: measure the audit description rewrite before shipping it #11540.
• Builds on fix(evals): warn on non-Skill denials in invocation-probe runs #11655 (order-aware denial rule).

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-01T00:15:14.153936Z	9a17653	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed 9a176535cbd3bacba5e730675a3ef10c59da7a2e: read 6 of 6 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #11755

I found no security issues.

What changed. The PR adds --ablation none --keep-temp --disallow-tools Read Glob Grep to the documented run line for emitted invocation-probe cases. It also adds two changelog fragments and one new test assertion. The change is in these files:

• the README text that measure-invocation.sh emit-plugin-eval writes
• reference/invocation-probes.md
• rewrite-a-plugin.md step 5
• measure-invocation.test.sh

What I checked.

• Shell script: the new lines in plugins/skill-quality/scripts/measure-invocation.sh sit inside a quoted heredoc (<<'EOF', L593). The shell writes them out as literal text, with no expansion or command substitution, so the change adds no way to inject commands.
• Tool grants: the change narrows what the eval agent can use. It removes Read, Glob and Grep and grants nothing new. The emitted allowed_tools line is unchanged.
• Removed instruction text: I read the deleted lines in invocation-probes.md under the instruction-surface lens. They only described which tools the emitted prompt.md allows. They were not a rule that guarded against anything, so removing them takes away no protection.
• Test: the new assertion only greps the generated README for the flag.
• Scope: the diff touches no trusted paths, trust controls, plugin manifests, workflows or token scopes.
• PR context file: it contains no embedded instructions aimed at the review lane.

Spend: none in this diff. The documented probe runs use prepaid model usage when someone runs them by hand.

critical-findings: 0

[github-actions]

claude-review reviewed 9a176535cbd3bacba5e730675a3ef10c59da7a2e: read 6 of 6 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #11755: zero findings

I read every hunk of all 6 changed files. None of them match a trusted-path glob.

What changed. Emitted invocation-probe runs now pass --ablation none --keep-temp --disallow-tools Read Glob Grep. The flag appears in three places:

• the README that measure-invocation.sh emit-plugin-eval writes, plus a new test that checks the flag is there;
• reference/invocation-probes.md under "emit-plugin-eval results";
• rewrite-a-plugin.md step 5.

There are two changelog fragments, both bump: patch.

Checks

• README heredoc. The README is written through a quoted heredoc (<<'EOF', measure-invocation.sh#L593). The shell therefore leaves the trailing \ on the run line alone, and the README shows it as a two-line command. The new grep -qF -- '--disallow-tools Read Glob Grep' test (measure-invocation.test.sh#L217) matches that text. The -- stops grep from reading the pattern as an option.
• Docs agree with each other. No other doc still says the emitted allowed_tools is what keeps exploratory reads from being denied. That claim was removed from invocation-probes.md. skills/check/SKILL.md and the emit-plugin-eval example in invocation-probes.md say nothing either way.
• Changelog bump. A patch bump fits: the change is to docs and generated README text, and the emitted case format is the same.
• Closing issue fix(skill-quality): stop invocation-probe runs failing validity on denied searches outside the workspace #11624, done-when ("An emitted probe suite run on a host returns VALID, or only warnings, under run-validity.py, with no change to the measured trigger rate beyond noise"). The PR's evidence for this is a host smoke run: 19 cases, VALID, 0 denials, 18/19 against 17/19 before. That result is author-claimed and I did not verify it; this lane cannot run a host eval. The new README test would fail if the flag were removed from the emitted run line, so the fix itself is protected against regression. The host outcome cannot be tested in CI.

Non-blocking note (not counted). The emitted prompt.md still lists allowed_tools: [Skill, Read, Glob, Grep], while the docs now tell users to remove those same tools with --disallow-tools. The PR body explains why: a run that forgets the flag keeps the milder denial pattern. invocation-probes.md does not say this, so a future reader may take the leftover grants for a mistake. A one-clause note there would prevent that.

Embedded text. Neither the PR context file nor the closing issue contains any instruction aimed at this lane.

Spend. None. The change only alters run flags in docs and the generated README.