Repository navigation
fix(skill-quality): stop invocation-probe runs failing validity on denied searches outside the workspace #11624
Description
Observed
With the #11618 fix, the 8 /ai-slop:audit trigger-measurement runs (measure-invocation.sh emit-plugin-eval cases, --ablation none --runs 3, Opus 5.5, Claude Code 2.1.316) no longer fail on misses or turn caps. All 8 are still INVALID on permission denials:
- The emitted cases set
allowed_tools: [Skill, Read, Glob, Grep]. The agent, finding an empty workspace, runs Glob and Grep at/home/kyle,~/.claude, the run's own temp workspace and the plugin copy's root. - The eval session denies them.
- A with-arm denial aimed at or under the plugin directory always FAILs in
run-validity.py.
Example: Glob /home/kyle/worktrees/ev6/a (the plugin copy root) in case ai-slop-audit-neg-train-01, run r1-a-pf.
The denials appear equally in both versions of the description, so they do not bias a before/after comparison. But skill-criteria.md Measurement step 4 reads numbers only from VALID runs, so no trigger measurement can pass today.
Fix wanted
One of these, chosen on evidence:
- (a) The emitter gives probe cases a workspace or tool set that leaves nothing to search outside it.
- (b)
run-validity.pytreats a denial in an invocation-probe case as a warning when it comes after the graded Skill decision. - (c) The emitted prompt names the workspace as the only place to look.
Measure the chosen fix on plugins/ai-slop/probes/audit.json.
Done when
- An emitted probe suite run on a host returns VALID, or only warnings, under
run-validity.py, with no change to the measured trigger rate beyond noise.
This was generated by AI.
Metadata
Metadata
Assignees
Labels
Type
Fields
No fields configured for Bug.
Activity
github-actions commentedon Oct 31, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: after the #11618 fix, all 8
/ai-slop:audittrigger-measurement runs are still INVALID. The agent searches outside the empty workspace, the eval session denies those searches, andrun-validity.pyfails any with-arm denial under the plugin directory. That leaves no trigger measurement able to pass. The issue lists three fixes (emitter workspace or tool set, a validity warning for post-decision denials, or a prompt naming the workspace) and asks for the choice to be made on evidence. I treated that as an implementation choice, not a design question. The done-when needs a host run, so a person or host run has to do it. No duplicate found; #11618 and #11540 are related context.Automated triage, applied by the lanes App.
kyle-sexton commentedon Oct 31, 2026
melodic-automation-lanes commentedon Oct 31, 2026
This records a re-fire of an earlier refusal, not a new refusal. The lanes kill switch stopped run 41219706363, the work-resolve-issue run that
agent-readystarted, before it did any work; that run ended at 2026-10-31T17:04:28Z. work-unblock has now removed and addedagent-readyback to start a fresh run. A kill-switch refusal is no fault in this issue or the lanes, so it never counts against later re-fires; after limits.limit-stops-per-issue of them since a person last addedagent-readyor a run last reported, work-unblock waits longer before each one, up to 24 hours.github-actions commentedon Oct 31, 2026
github-actions commentedon Oct 31, 2026
already-leased: another session holds a live lease on #11624, so run https://github.com/melodic-software/claude-code-plugins/actions/runs/41236209491 ran no model.
kyle-sexton commentedon Oct 31, 2026
kyle-sexton commentedon Oct 31, 2026
Status: the denial rule shipped (#11655); the probe-setup fix is still open. One option tested and ruled out.
#11655 makes the denial check order-aware: a probe denial warns only after the graded Skill call, or on a should-not-fire hold whose denial-free runs also held. Under that rule 2 of the 8 #11540 runs gate VALID. The other 6 have the agent search outside its empty workspace, get denied, and then decide.
Option (a) by tool set is ruled out. I emitted the
/ai-slop:auditprobes withallowed_tools: [Skill]and ran 19 cases × 1 with-arm run (Claude Code 2.1.316, Opus 5.5). The agent still called Glob, Read and Grep, and now every call was denied, including Globs of its own run home. That made 16 of 19 runs denied, against 6 of 8 runs before. Per the docs,allowed_toolsand--allow-toolsadd grants beyond the read-only set and never remove it:(both read 2026-10-31). The change was not pushed.
Options left:
Either one needs a paid measurement. Parked under the owner's 2026-10-31 usage plan: build the next attempt, check it on these 19 cases with
--runs 1, and batch the full #11540 measurement with the otherhost-runitems.