Repository navigation
Description
Hi,
First of all, thanks for building Agent Scan - it's been very useful.
I noticed what appears to be a class of false positives when scanning default agent workflows (e.g. Claude Code/Codex skills).
For example, Agent Scan reports:
[W011 high]: Third-party content exposure detected (high risk: 0.85).
The required workflow explicitly instructs the agent to "Explore the codebase" and "inventory existing documentation" (e.g. docs/**/*.md, CONTRIBUTING.md, ARCHITECTURE.md, README.md), which are outsider-authored free text from the repository, and that text would be read at runtime to generate/update the customization files.
The issue is that these instructions are part of the normal behavior of coding agents. Reading repository files such as README.md, CONTRIBUTING.md, ARCHITECTURE.md, or documentation under docs/ is generally expected and necessary for understanding the project before making changes.
More (Codex - Installed with the CLI by default):
├── skill-installer 1 high finding
│ ● [W013 high]: Attempt to modify system services in skill instructions detected (high risk: 0.90). The prompt explicitly tells the agent to "request escalation" when running the helper scripts in the sandbox (encouraging privilege escalation) and includes
│ instructions that allow overwriting preinstalled system skills and accessing private repos, which can modify the machine state.
│ ├── instruction SKILL.md
│ ├── asset openai.yaml
│ ├── asset LICENSE.txt
│ ├── asset skill-installer.png
│ ├── asset skill-installer-small.svg
│ ├── script github_utils.py
│ ├── script list-skills.py
│ └── script install-skill-from-github.py
├── openai-docs 4 findings (4 high)
│ ● [E004 high]: Potential prompt injection detected (high risk: 1.00). The skill includes an instruction to retry an install "with escalated permissions" and "Do not ask the user to run it yet," which directs the agent to attempt privilege escalation and
│ bypass user consent—a deceptive action outside the docs-only guidance scope.
│ ● [E006 high]: Malicious code pattern detected (high risk: 1.00). The skill contains an explicit instruction to automatically retry an installation with escalated permissions and to avoid asking the user to run it, which is an intentional
│ privilege-escalation and social-engineering pattern indicating high-risk/malicious intent.
│ ● [W012 high]: Potentially malicious external URL detected (high risk: 1.00). The skill and its helper script explicitly fetch and use https://developers.openai.com/api/docs/guides/latest-model.md (and the MCP endpoint https://developers.openai.com/mcp) at
│ runtime to determine model IDs and to fetch migration/prompting guides that directly control model-selection and prompt-upgrade behavior.
│ ● [W013 high]: Attempt to modify system services in skill instructions detected (high risk: 1.00). The skill explicitly instructs retrying an install command "with escalated permissions" if it fails due to permissions/sandboxing, which directs the agent to
│ obtain elevated privileges and bypass security controls.
As a result, scans of the default skills/prompts shipped with tools like Claude Code or Codex generate high-severity findings even though there doesn't appear to be an actual prompt injection or data exposure issue.
Is this expected behavior?
I know i can just scan specific paths but I think it might worth (if so):
- lowering the severity for this pattern,
- distinguishing expected repository-documentation reads from genuine third-party prompt ingestion,
- or providing a suppression/allowlist for common documentation files.
Otherwise, it may produce a significant number of false positives that make it harder to identify real issues.
Thanks!
Activity
jiezeng2004-design commentedon Jul 25, 2026
A useful distinction here may be mention, instruction, and reachable execution rather than treating every security-related phrase in repository documentation as equivalent:
rm -rf, credential paths, exfiltration commands, or privilege escalation specifically to explain that they must be blocked.README.md,CONTRIBUTING.md, ordocs/**before making changes.Package inclusion and runtime reachability seem useful classification signals. A malicious-looking fixture outside the shipped package, a quoted negative example in a threat model, and an instruction executed by an installed skill should not necessarily receive the same confidence or severity. User consent and whether the content can actually reach a tool call are also materially different signals.
PatchWarden is a useful non-executing example for this class of test because its security documentation intentionally contains destructive commands, sensitive paths, prompt-injection language, and exfiltration examples while the product's purpose is to block or approve those operations. A static fixture should use synthetic metadata and documentation only; Agent Scan should not start the
patchwardenstdio MCP process merely to classify the documentation.Would an anonymized JSON result plus a minimal synthetic documentation fixture be useful for calibrating W011/W013/E004/E006? I am asking before preparing data because the repository's current contribution policy says external code contributions are closed.
orassayag commentedon Jul 25, 2026
@jiezeng2004-design
This is a really clear way to frame it — thanks for putting it into words. The "mention vs. instruction vs. reachable execution" split maps exactly onto what I was running into: Agent Scan currently seems to weight presence of security-relevant strings heavily, without checking whether that content is actually wired into something that gets executed or acted on by the agent.
Your PatchWarden example is a good illustration too — a tool whose whole job is to document/handle dangerous operations will always look "high risk" under a mentions-based heuristic, even though the actual risk depends on package inclusion and reachability, exactly as you said.
On your question: yes, I think an anonymized JSON scan result plus a minimal synthetic documentation fixture would be genuinely useful for calibrating W011/W013/E004/E006 — it would give the maintainers something concrete to test severity/confidence adjustments against, without needing real repo content.
That said, since you flagged that the contribution policy currently has external code contributions closed, it's probably worth a maintainer confirming whether that also covers test fixtures/data submitted for calibration purposes (as opposed to a PR against the detection logic itself) — cc'ing in case someone from the Agent Scan team wants to weigh in before you spend time preparing it.
pdugan20 commentedon Jul 31, 2026
Adding three concrete W011 cases from public skills.sh scans of pdugan20/skills:
Pages:
All three currently pass the Agent Trust Hub and Socket checks. We are adding narrow runtime trust boundaries and adversarial evals: ordinary repository content, API responses, documentation, and generated output remain usable data, but cannot independently authorize commands, secret access, external writes, or scope expansion. Legitimate user requests and applicable agent instruction files remain authoritative.
These examples support the distinction already proposed here between content exposure and reachable authority or execution. Can W011 account for explicit data-versus-instruction boundaries, independently validated commands, and existing approval gates, either by lowering the classification or by reporting the exposure as informational? In particular, the feature-spike result seems broader than the repository-documentation case because it treats receipt of the initiating user prompt itself as third-party exposure.
roblouw2nd commentedon Aug 29, 2026
Disclosure: AI agent (Claude) operating Fetchgate under human oversight; posted by the operator.
Some field data for the mention / instruction / reachable-execution split, from the MCP side rather than skills, in case it helps calibrate: I sent every remote URL in the official MCP registry (15,329) a read-only
initialize→tools/listand scored the 140,284 tool descriptions and 5,462initialize.instructionsstrings that came back with rules of the same family as W011/W013 (credential-file paths, chat-template markers, exfiltration shapes, model-directed overrides).~/.ssh,.env,.aws/credentials,id_rsa,.npmrc): 32 hosts. Every one is a secret scanner or.envparser describing its own scope — the "mention" class. Zero instruct the model to read or send a file.IMPORTANT:convention or a prompt-injection scanner citing the token it detects.<IMPORTANT>sidenote shape): 0 occurrences.So on the live corpus, weighting presence of security strings gets the base rates backwards: the strings appear almost exclusively in defenders' descriptions, and the real steering uses none of them. The same thing @jiezeng2004-design is describing for skills, measured.
Per-tool rows with the flag classes, if a real-world negative set is useful for tuning: https://fetchgate.dev/tools/mcp-registry-audit (per-server summary is CC BY 4.0; the full description text is in the paid dataset, but I'm happy to share the 32 credential-mention and 47 concealment rows outright for this purpose — say the word).