Skip to content

[evoskill] Add restricted-tool-triage skill for gh-aw agent tool-denial handling #53010

Description

@github-actions

Caution

Protected files were modified in this change.
This pull request is in request_review mode and requires explicit human scrutiny before merge.

Protected files: .github/skills/restricted-tool-triage/SKILL.md

EvoSkill: New skill restricted-tool-triage

Diagnosed capability gap: Agents running gh-aw workflows with a narrow tools:/bash allowlist have no procedural guidance for handling tool-denial events. When denied, they tend to retry variants of the same disallowed command until hitting the harness's hard denial-count threshold (commonly 3), which aborts the entire session. They also sometimes misreport an intentionally scoped-down toolset as a credentials/infra problem via missing_tool.

Training evidence: Code Scanning Fixer run 31919288964 — workflow restricts tools.bash to ["git diff:*", "git restore:*", wc]. Agent tried sed -n ... file.js (denied), read(/tmp/.../file.js) (denied), then git status --short\ngit diff --stat (denied, since only git diff/git restore are allowed) — hit 3/3 denials, guard.tool_denials_exceeded fired, session aborted. The agent's missing_tool call then blamed "token scopes, repository permissions" — misdiagnosing an intentional least-privilege config as an infra bug.

Three other failures reviewed in the same training window (Smoke DeepSeek Harness npm ETARGET version-pinning error, Agentic Commands route job "Argument list too long" argv-size bug, Design Decision Gate missing workflow_dispatch pr_number input) were determined to be infrastructure/code bugs, not skill-addressable, and were excluded from the proposal.

Mutation type: Create (new, non-overlapping skill — no existing skill under .github/skills/ covers restricted-toolset triage for the agent's own behavior).

Validation (held-out, 5 samples spanning Code Scanning Fixer success case, Daily Firewall Logs Collector success, PR Sous Chef unrestricted-toolset success, Contribution Check restricted-toolset failure, Test Quality Sentinel single-denial-then-correct-pivot success):

  • Baseline score: 55/100
  • Candidate score: 82/100
  • Regressions: none
  • Confidence: high

The candidate generalizes beyond the training example (different command families: git log/show/status/cat, go build/vet) without disrupting successful restricted-toolset runs or unrestricted-toolset runs, and adds explicit guardrails against mislabeling an intentional scope restriction as a permissions/credentials bug.

Note: Only .github/skills/restricted-tool-triage/SKILL.md was added. The underlying model and all other repository files (workflows, source code, root instructions) are unchanged.

Generated by 🧬 EvoSkill Evolver · auto · 144.7 AIC · ⌖ 5.28 AIC · ⊞ 8.4K ·

  • expires on Aug 22, 2026, 6:48 PM UTC-08:00

Note

This was originally intended as a pull request, but the git push operation failed.

Original error: ERR_VALIDATION: pushSignedCommits: refusing unsigned push for branch 'evoskill/restricted-tool-triage-070c1acb5f1fd869': Signed-commit payload violates file-protection policy (request_review): .github/skills/restricted-tool-triage/SKILL.md

Workflow Run: View run details and download bundle artifact

The bundle file is available in the agent artifact in the workflow run linked above.

To create a pull request with the changes:

# Download the artifact from the workflow run
gh run download 31922222282 -n agent -D /tmp/agent-31922222282

# Fetch the bundle into a temporary ref, then update the local branch
git fetch /tmp/agent-31922222282/aw-evoskill-restricted-tool-triage.bundle refs/heads/evoskill/restricted-tool-triage:refs/bundles/create-pr-evoskill-restricted-tool-triage-070c1acb5f1fd869-93a124c3
git update-ref refs/heads/evoskill/restricted-tool-triage-070c1acb5f1fd869 refs/bundles/create-pr-evoskill-restricted-tool-triage-070c1acb5f1fd869-93a124c3
git checkout evoskill/restricted-tool-triage-070c1acb5f1fd869
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d refs/bundles/create-pr-evoskill-restricted-tool-triage-070c1acb5f1fd869-93a124c3

# Push the branch to origin
git push origin evoskill/restricted-tool-triage-070c1acb5f1fd869

# Create the pull request
gh pr create --title '[evoskill] Add restricted-tool-triage skill for gh-aw agent tool-denial handling' --base main --head evoskill/restricted-tool-triage-070c1acb5f1fd869 --repo github/gh-aw

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions