Skip to content

Improve PR Agent Gate verification to prevent result fabrication #33806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PureWeen merged 2 commits into from
Jan 30, 2026
Merged

Improve PR Agent Gate verification to prevent result fabrication #33806

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f8ce7c3
Improve PR agent Gate verification workflow
PureWeen Jan 30, 2026
6112c1f
Trim Gate documentation to reference skill instead of duplicating
PureWeen Jan 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 29 additions & 2 deletions .github/agents/pr.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -441,10 +441,23 @@ Tests were already verified to FAIL in Phase 2. Gate is a confirmation step:
**If starting from a PR (fix exists):**
Use full verification mode - tests should FAIL without fix, PASS with fix.

```bash
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -RequireFullVerification
**🚨 MUST invoke as a task agent** to prevent command substitution:

```markdown
Invoke the `task` agent with agent_type: "task" and this prompt:

"Invoke the verify-tests-fail-without-fix skill for this PR:
- Platform: android (or ios)
- TestFilter: 'IssueXXXXX'
Comment thread
PureWeen marked this conversation as resolved.
- RequireFullVerification: true

Report back: Did tests FAIL without fix? Did tests PASS with fix? Final status?"
```

**Why task agent?** Running inline allows substituting commands and fabricating results. Task agent runs in isolation and reports exactly what happened.

See `.github/skills/verify-tests-fail-without-fix/SKILL.md` for full skill documentation.

### Expected Output (PR with fix)

```
Expand Down Expand Up @@ -493,3 +506,17 @@ pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1
- ❌ **Running tests during Pre-Flight** - That's Phase 3
- ❌ **Not creating state file first** - ALWAYS create state file before gathering context
- ❌ **Skipping to Phase 4** - Gate MUST pass first

## Common Gate Mistakes

- ❌ **Running Gate verification inline** - Use task agent to prevent command substitution
- ❌ **Using `BuildAndRunHostApp.ps1` for Gate** - That only runs ONE direction; the skill does TWO runs
- ❌ **Using manual `dotnet test` commands** - Doesn't revert/restore fix files automatically
- ❌ **Claiming "fails both ways" from a single test run** - That's fabrication; you need the script's TWO runs
- ❌ **Not waiting for task agent completion** - Script takes 5-10+ minutes; wait for task to return

**🚨 The verify-tests-fail.ps1 script does TWO test runs automatically:**
1. Reverts fix → runs tests (should FAIL)
2. Restores fix → runs tests (should PASS)

Never run Gate inline. Always invoke as task agent.
3 changes: 2 additions & 1 deletion .github/skills/ai-summary-comment/scripts/post-ai-summary-comment.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,8 @@ function Extract-AllSections {
$sections = @{}

# Pattern to find all <details><summary><strong>TITLE</strong></summary>...content...</details> blocks
$pattern = '(?s)<details>\s*<summary><strong>([^<]+)</strong></summary>(.*?)</details>'
# Note: [^>]* handles optional attributes like "open" in <details open>
$pattern = '(?s)<details[^>]*>\s*<summary><strong>([^<]+)</strong></summary>(.*?)</details>'
$matches = [regex]::Matches($StateContent, $pattern)

if ($Debug) {
Expand Down