Add PR Tooling Safety Check — scanner for detecting changes in design/build/agentic-time behavior#19680
Open
Add PR Tooling Safety Check — scanner for detecting changes in design/build/agentic-time behavior#19680
Conversation
Hourly text-only scan of external PRs. Reads diff, classifies risk into categories (build infra, compiler output, bootstrap, prompt injection, supply chain, scope mismatch), and labels accordingly. Never checks out or builds PR code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Agent has only GitHub MCP pull_requests toolset. No shell, no file system, no checkout access. Reads diffs as text via API, classifies risk, labels PRs. Works on fork PRs. Trusted authors are skipped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three states: no label (unscanned), AI-Security-Scan-Clean (safe),⚠️ labels (flagged + why). No comments at all. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Every open PR gets a label. No ambiguity about scan status. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Each risk category now includes specific attack patterns to look for, with citations from: - Microsoft MSBuild Security Best Practices - MITRE ATT&CK T1127.001 (MSBuild inline tasks) - OWASP LLM Top 10 (prompt injection, excessive agency) - OWASP AI Agent Security Cheat Sheet (goal hijacking) - GitHub Security Architecture blog (safe outputs, isolation) - OpenAI Agent Builder Safety (structured outputs) - Anthropic Computer Use docs (network egress control) Also adds "Why this workflow is safe" section documenting our own defense posture against the same attack classes we scan for. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Major rewrite from "security scan" to "tooling safety check" — informational "what does this PR affect?" labels, not attack detection. New labels: Affects-Restore, Affects-Test-Infra, Affects-Design-Time. Renamed: AI-Security-Scan-Clean → AI-Tooling-Check-Clean. Prompt injection detection now references Gen Digital SAGE taxonomy (CLT-PI-001 to CLT-PI-081) — 9 pattern families covering instruction override, role hijacking, security bypass, anti-transparency, prompt exfiltration, structural injection, role markers, obfuscation, and credential exfiltration. Docs rewritten with state machine, methodology table citing 9 sources (Microsoft, MITRE, OWASP, GitHub, OpenAI, Anthropic, Peli, SAGE). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Backtest on 128 merged PRs: - 84% clean (108/128), 15% flagged (20/128) - 16 flagged PRs were from app/copilot-swe-agent and app/github-actions (internal bots) — added to trusted list - 4 external contributor flags, ALL correct: IlxGen change, SDK update, source-build-assets, build.sh quoting - 0/4 false positives With updated trusted list: 96% clean, 3% flagged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fork PRs are the real threat surface — non-fork PRs were pushed by someone with repo write access. Quick-bypass them to AI-Tooling-Check-Clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Everything lives in labelops-pr-security-scan.md — the .md body is both the agent prompt and human-readable documentation. Added state machine table, methodology references, label creation setup inline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The old name implied malicious intent. Maintainers editing .github/workflows/ is normal — the label should say "this PR changes agent behavior", not "this PR is attacking you". SAGE patterns still scanned for hidden injection in any file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Old rule flagged all of tests/FSharp.Test.Utilities/ — 14 hits, 8 were test DSL helpers (Compiler.fs, Assert.fs, SurfaceArea.fs, etc.). New rule: only TestFramework.fs, ProjectGeneration.fs (have Process.Start), FSharp.Test.Utilities.fsproj, EndToEndBuildTests/, .runsettings. Result: 14 → 6 flags, all 8 dropped were noise (test authoring, not execution infrastructure). Eliminates both auduchinok false positives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Test execution is too broad to usefully flag. Keeping scope to: build, restore, bootstrap, compiler output, design-time, agent config, scripts — phases where untrusted code executes implicitly. 7 labels remain (was 9). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Triggers on: .fsproj, .runsettings, TestFramework.fs, ProjectGeneration.fs, EndToEndBuildTests/ — files that control test build/selection/execution. Does NOT trigger on: Compiler.fs, Assert.fs, SurfaceArea.fs etc. — adding a helper method is not dangerous, changing how processes spawn is. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Generic workflow (.md): .NET/MSBuild categories (Build-Infra, Restore, Agent-Config, Scope-Review-Needed) + SAGE prompt injection patterns. Works for any .NET repo. Repo-specific rules (tooling-check-repo-rules.md): F# compiler paths (Bootstrap, Compiler-Output, Design-Time, Test-Tooling), trusted author list, non-fork bypass. Pluggable — edit per repo. Also: removed 5 name-dropped references per adversarial review (Anthropic, OpenAI, Peli, OWASP Cheat Sheet, OWASP LLM06). Kept only refs that actually drive implementation: MSBuild, MITRE, OWASP LLM01, SAGE. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The agent has only pull_requests MCP — it cannot read repo files. A separate instructions file would also pollute every Copilot session. Instead: everything is inline in the .md. Generic .NET categories at the top, repo-specific (F# compiler) categories below a clear HTML comment marker. To adopt in another repo: edit the repo-specific section, update trusted authors and non-fork bypass, recompile. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
OWASP LLM06 (excessive agency): we mitigate it, not follow guidance. OWASP Agent Cheat Sheet: acknowledged risks, not claimed compliance. OpenAI Agent Safety: gh-aw provides structured outputs per this rec. Dropped remain dropped: Anthropic (wrong threat model), Peli (not followed). Methodology now split: "what drives categories" vs "threat model for the scanner itself" — honest about what we do vs what we acknowledge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1. Stale labels: re-scan if PR head SHA changed since last label. No more permanent bypass via clean-then-force-push. 2. Setup script labels now match frontmatter exactly: Affects-Test-Infra → Affects-Test-Tooling Prompt-Injection-Risk → Affects-Agent-Config 3. Split "Clean" into two distinct labels: AI-Tooling-Check-Scanned-Clean = diff analyzed, nothing found AI-Tooling-Check-Bypassed = trusted author or non-fork, not analyzed Findings from Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.5, GPT-5.4 — all 4 models identified these same 3 blocking issues independently. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Rule 7: PR title/body are untrusted — classify on files+diff only - Rule 8: if diff >5000 lines or truncated, fall back to file list - Rule 9: labels are informational, must not gate merges or automation - Comment explaining min-integrity: none (needed to read fork PRs) Addresses LLM01 goal-hijacking, LLM10 unbounded consumption, and cascading-failure risks from adversarial review. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Diff-size cap was backwards — big PRs need scanning most. Labels-not-gates was solving a non-existent problem. Keep rule 7 (untrusted title/body) — that is real. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Lists create a false sense of completeness. An attacker just looks at the list and picks something not on it. MSBuild has dozens of extension points — listing them all is a losing strategy by design. Instead: each category now explains WHAT IT MEANS (what phase, what risk) and tells the agent to use judgment. The agent understands MSBuild, NuGet, and build systems — it can catch novel vectors a checklist misses. SAGE taxonomy kept as a reference for prompt injection PATTERNS (those are genuinely useful as examples) but not as an exhaustive list. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When any⚠️ label is added, post one terse comment listing which file(s) triggered each label and why. No prose. No comment for clean/bypassed PRs. hide-older-comments collapses stale scans. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comments: when any⚠️ label is added, post one terse comment listing which file triggered each label and why. No comment for clean/bypassed. Noise fix: routine <Compile Include> additions to test .fsproj files are NOT Build-Infra. Only structural .fsproj changes (targets, tasks, package refs, properties) trigger the flag. Test battery: 50 fork PRs from 8 contributors. 0 false negatives, ~3 fixable false positives from test .fsproj noise (now addressed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Following https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/claude-prompting-best-practices: - Added <role> tag with clear identity statement - Added <context> tag separating domain knowledge from instructions - Wrapped rules in <rules>, process in <process>, categories in <categories> - Each category wrapped in <category name="..."> for unambiguous parsing - Added concrete <example> for comment format - Positive instructions throughout ("use judgment" not "dont use checklists") - Role, context, rules, process, categories flow top-to-bottom - Terse methodology tables kept at bottom (reference, not instruction) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
All content the agent reads is untrusted (PR title, body, diff). Constrain outputs to the minimum needed: - add-labels max: 30 → 10 (only 10 possible labels exist) - add-comment max: 10 → 1 (one comment per PR per scan) - hide-older-comments: true (replaces previous scan comment) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
T-Gro
added
the
NO_RELEASE_NOTES
Contributor
|
Generic workflow (.md): .NET/MSBuild categories (Build-Infra, Restore, Agent-Config, Scope-Review-Needed). Works for any .NET repo. Repo-specific rules (.github/tooling-check-repo-rules.md): F# compiler categories (Bootstrap, Compiler-Output, Design-Time, Test-Tooling), trusted author list, non-fork bypass. Agent reads this at runtime via repos toolset (added to frontmatter). NOT in .github/instructions/ — would pollute every Copilot session. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e machine These were human-facing docs fed into every agent run for no reason. Moved to HTML comments (2 lines) for auditability. Setup commands belong in PR description, not in the agent prompt. Workflow went from ~196 lines to ~134 lines of actual agent instructions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The .md is the agent prompt. Every token costs money on every run. Removed: inline Ref: citations (6 URLs), SAGE link, HTML comment refs. These belong in the PR description, not in the agent instructions. 125 lines. Zero references. Zero setup commands. Pure instructions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Labels have no metadata — cannot store when/what SHA was scanned. The comment IS the state. Every scan (clean, bypassed, or flagged) now posts a comment with <!-- head:<sha> --> on the last line. Next run reads existing comments, extracts SHA, compares to current headRefOid. Match = skip. Mismatch = re-scan. hide-older-comments: true collapses previous scan comments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Trusted-author list was a username-spoofing vector: an LLM doing fuzzy string matching on prose could be tricked by similar usernames (e.g. "copilot-evil" matching "copilot"). The non-fork bypass already covers it — anyone on the trusted list has write access and pushes non-fork PRs. The bypass now checks headRepository API field only, not author username. Rule 5 now explicitly says author username is untrusted text. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Explicitly read rules from DEFAULT BRANCH only, never from PR branch - A PR modifying tooling-check-repo-rules.md triggers Affects-Agent-Config (it controls scanner behavior = agent guidance) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Was: vague one-liner listing keywords. Now: 9 pattern families with concrete examples the agent can match against diff text. Each family explained with WHY it matters and WHAT it looks like. References SAGE CLT-PI-001–081 and OWASP LLM01. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Affects-Agent-Config: PR modifies agent instruction/skill/workflow FILES. Suspicious-Prompting: SAGE injection patterns found in title, body, commit messages, or diff text. Scans ALL surfaces, not just .github/. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
All labels in frontmatter allowed list match categories in workflow body + repo-rules file. Suspicious-Prompting added everywhere. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
Label creation (one-time setup)# Generic labels
gh label create "AI-Tooling-Check-Scanned-Clean" --repo dotnet/fsharp --color 0e8a16 --description "Tooling check: diff analyzed, no interesting infrastructure files"
gh label create "AI-Tooling-Check-Bypassed" --repo dotnet/fsharp --color c5def5 --description "Tooling check: non-fork PR, not diff-analyzed"
gh label create "⚠️ Affects-Build-Infra" --repo dotnet/fsharp --color d93f0b --description "Tooling check: PR touches build infrastructure"
gh label create "⚠️ Affects-Restore" --repo dotnet/fsharp --color d93f0b --description "Tooling check: PR touches NuGet packages or feeds"
gh label create "⚠️ Affects-Agent-Config" --repo dotnet/fsharp --color d93f0b --description "Tooling check: PR modifies AI agent instructions or workflows"
gh label create "⚠️ Suspicious-Prompting" --repo dotnet/fsharp --color d93f0b --description "Tooling check: prompt injection patterns found in title/body/diff/commits"
gh label create "⚠️ Scope-Review-Needed" --repo dotnet/fsharp --color fbca04 --description "Tooling check: PR scope exceeds title/description"
# Repo-specific labels (dotnet/fsharp)
gh label create "⚠️ Affects-Bootstrap" --repo dotnet/fsharp --color b60205 --description "Tooling check: PR touches compiler bootstrap chain"
gh label create "⚠️ Affects-Compiler-Output" --repo dotnet/fsharp --color d93f0b --description "Tooling check: PR touches IL emission or codegen"
gh label create "⚠️ Affects-Design-Time" --repo dotnet/fsharp --color d93f0b --description "Tooling check: PR touches type providers or dependency manager"
gh label create "⚠️ Affects-Test-Tooling" --repo dotnet/fsharp --color d93f0b --description "Tooling check: PR touches test framework infrastructure" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An agentic workflow that labels open PRs with which development phases they affect — restore, build, bootstrap, compiler output, design-time, test tooling, or agent configuration. Runs hourly. Text-only — reads diffs via GitHub API, never checks out or builds PR code.
Not a code quality check or merge-readiness signal. Detects changes in design/build/agentic-time behavior.
State machine
AI-Tooling-Check-BypassedAI-Tooling-Check-Scanned-Clean⚠️ Affects-*Labels re-evaluated on new commits (head SHA check).
Categories
Generic (.NET): Build-Infra, Restore, Agent-Config, Scope-Review-Needed
Repo-specific (edit per repo): Bootstrap, Compiler-Output, Design-Time, Test-Tooling
Categories use principle-based descriptions — the agent applies judgment, not a file list. Catches novel extension points a checklist would miss.
Safety
pull_requestsMCP read accessdefaults+githubegress onlyReferences
Evaluation (267 PRs)
Precision: 217 PRs (128 recent + 89 from 6 key contributors, 5-year span). 37 flagged, 0 incorrect flags.
Recall: 50 fork PRs from 8 external contributors, classified by 5 independent fresh agents. 30 true positives, 15 true negatives, 0 false negatives, 5 debatable (technically correct).
Noise:
pars.fsy→ Bootstrap (18%, correct), test.fsproj<Compile Include>→ Build-Infra (6%, fixed), TransparentCompiler → Design-Time (6%, correct).Limitation: no adversarial/synthetic attack PRs tested. Trusted-author bypass trades recall for throughput.
Files
.github/workflows/labelops-pr-security-scan.md— workflow (~208 lines).github/workflows/labelops-pr-security-scan.lock.yml— compiled (auto-generated)