[CI] Extend gate to all test types and decouple from PR review

[kubaflo]

Summary

Extends the CI PR review pipeline to support all test types (UI tests, device tests, unit tests, XAML tests) and restructures the review flow by decoupling the gate from the copilot agent.

Before

• Gate only supported UI tests (TestCases.HostApp / TestCases.Shared.Tests)
• PRs with device tests, unit tests, or XAML tests were skipped by the gate
• Gate ran as Phase 2 inside the copilot agent (4-phase: Pre-Flight → Gate → Try-Fix → Report)
• Gate results were duplicated across all phase outputs
• AI summary comment included session history merging (841 lines of code)

After

• Gate supports all test types with auto-detection
• Gate runs as a standalone script step before the copilot agent
• Gate posts its own separate PR comment (<!-- AI Gate -->)
• AI summary is simplified (170 lines, always overwrites, no session history)
• PR review is now 3 phases: Pre-Flight → Try-Fix → Report

New Scripts

Script	Purpose
Detect-TestsInDiff.ps1	Analyzes PR files, classifies tests by type (UITest, DeviceTest, UnitTest, XamlUnitTest), extracts method names from diffs
post-gate-comment.ps1	Posts/updates gate result as separate PR comment
RunTests.ps1	Unified test runner entry point for all test types

Test Detection

pwsh .github/scripts/shared/Detect-TestsInDiff.ps1 -PRNumber 25129

📱 [DeviceTest] EditorTests (PlaceholderHorizontalTextAlignment)
   Filter:  Category=Editor
🖥️ [UITest] Issue10987
   Filter:  Issue10987

New Review Flow

Step 0: Branch setup
Step 1: Gate (verify-tests-fail.ps1 — direct script, no copilot agent)
         → Posts <!-- AI Gate --> comment immediately
Step 2: PR Review (copilot agent — 3 phases: Pre-Flight, Try-Fix, Report)
         → Gate result passed in prompt
Step 3: Post AI Summary (<!-- AI Summary --> comment)
Step 4: Apply labels

PR Comments (Two Separate Comments)

Gate comment (<!-- AI Gate -->):

## 🚦 Gate — Test Verification
► Expand Full Gate — abc1234 · Fix editor alignment

### Gate Result: ✅ PASSED
| Step | Expected | Actual | Result |
| Without fix | FAIL | FAIL | ✅ |
| With fix | PASS | PASS | ✅ |

AI Summary comment (<!-- AI Summary -->):
Pre-Flight, Fix, Report sections only — no gate duplication.

Key Changes

• verify-tests-fail.ps1: Auto-detects test type, routes to correct runner (BuildAndRunHostApp, Run-DeviceTests, dotnet test), iterates over all detected tests, -Platform mandatory
• Detect-TestsInDiff.ps1: Shared detection engine — reads [Category] attributes for device test filtering, extracts method names from PR diffs
• Review-PR.ps1: Gate as Step 1 (script), PR review as Step 2 (copilot), removed PR finalize step
• post-ai-summary-comment.ps1: Rewritten from 841 → 170 lines, always overwrites
• pr-gate.md: Strict output template, no cross-phase duplication rule
• pr-review/SKILL.md: 3 phases (removed Gate), no-duplication rule
• EstablishBrokenBaseline.ps1: Excludes TestUtils/DeviceTests.Runners from fix file detection

Verified

• Gate passed locally on Share device tests: without fix=FAIL ✅, with fix=PASS ✅
• Detection tested on PRs: Fixed Editor HorizontalTextAlignment does not update at run time #25129, [Testing] Refactoring Feature Matrix UITest Cases for Editor Control #34615, [iOS, MacCatalyst] Fix CollectionView grid spacing updates for first row and column #34598, [Net10] OnSizeAllocated in Shell not triggered - fix #31056
• Comments posted to 8 PRs from CI build artifacts

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34705

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34705"

[Copilot]

Pull request overview

Extends the CI “gate” and PR review automation to detect and run all MAUI test types (UI/device/unit/XAML), and restructures the review flow so gate runs as a standalone step with its own PR comment, while the Copilot-driven review focuses on Pre-Flight/Try-Fix/Report.

Changes:

• Update verify-tests-fail.ps1 to auto-detect test type(s) and dispatch to the right runner, running all detected tests.
• Add shared test detection (Detect-TestsInDiff.ps1) and a dedicated gate comment poster (post-gate-comment.ps1); simplify AI summary posting.
• Update review orchestration/docs to remove gate from the pr-review skill and run it from Review-PR.ps1 instead.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1	Adds multi-test-type detection/routing and multi-test execution for gate verification.
.github/skills/verify-tests-fail-without-fix/SKILL.md	Updates skill documentation for broader test-type support (but currently out of sync with script behavior/outputs).
.github/skills/try-fix/references/example-invocation.md	Documents device/unit test command examples.
.github/skills/try-fix/SKILL.md	Updates try-fix guidance to select the correct runner per test type.
.github/skills/pr-review/SKILL.md	Changes orchestrator to 3 phases (Pre-Flight/Try-Fix/Report) and states gate is pre-run.
.github/scripts/shared/Detect-TestsInDiff.ps1	New shared test detection/classification script used by gate and tooling.
.github/scripts/post-gate-comment.ps1	New script to post/update a dedicated <!-- AI Gate --> PR comment.
.github/scripts/post-ai-summary-comment.ps1	Simplifies AI summary comment generation (no session history; gate posted separately).
.github/scripts/RunTests.ps1	New unified local entry point to run Unit/Device/UI/Integration tests.
.github/scripts/Review-PR.ps1	Reorders flow to run gate first via script, then invoke pr-review, then post comments/labels.
.github/scripts/EstablishBrokenBaseline.ps1	Expands “test path” patterns to exclude more test utility/runner paths from fix detection.
.github/pr-review/pr-report.md	Updates report phase prerequisites now that gate is external to pr-review.
.github/pr-review/pr-gate.md	Updates gate doc for detection template/output rules (but still references task-agent flow).
.github/copilot-instructions.md	Updates repository Copilot instructions to reflect 3-phase PR review and multi-type gate.

[Copilot]

Log file names are derived directly from TestName (e.g., test-failure-<TestName>.log). TestName can include spaces/parentheses/commas (device tests append method names), which can create awkward or invalid paths on some filesystems and can hit path-length limits. Consider sanitizing TestName for file names (or use an index-based file name) and store the display name inside the log content instead.

Suggested change

	$TestLog = Join-Path $OutputPath "test-failure-$($testEntry.TestName).log"
	# Sanitize TestName for use in a file name and keep it reasonably short
	$rawTestName = [string]$testEntry.TestName
	$invalidFileNameChars = [IO.Path]::GetInvalidFileNameChars()
	$extraProblematicChars = [char[]]' ()[],'
	$charsToReplace = $invalidFileNameChars + $extraProblematicChars
	$sanitizedTestName = ($rawTestName.ToCharArray() | ForEach-Object {
	if ($charsToReplace -contains $_) { '_' } else { $_ }
	}) -join ''
	if ([string]::IsNullOrWhiteSpace($sanitizedTestName)) {
	$sanitizedTestName = "test-$testIndex"
	}
	$maxNameLength = 60
	if ($sanitizedTestName.Length -gt $maxNameLength) {
	$sanitizedTestName = $sanitizedTestName.Substring(0, $maxNameLength)
	}
	$TestLog = Join-Path $OutputPath ("test-failure-{0}.log" -f $sanitizedTestName)

[Copilot]

This doc claims the script "Updates PR labels" as part of the workflow, but the updated verify-tests-fail.ps1 no longer contains label-management logic. Please remove or update these steps so they reflect the current behavior (labeling appears to happen later in Review-PR.ps1).

[Copilot]

For each device-test file, Step 4 calls gh api repos/dotnet/maui/pulls/$PRNumber/files and filters it with jq. This creates an N+1 GitHub API pattern that can hit rate limits and slow down gate on large PRs. Consider fetching the PR files/patches once (single API call) and caching them in a lookup keyed by filename.

[Copilot]

When running without -PRNumber, Step 4 uses git diff $mergeBase HEAD -- $file, but $mergeBase is only set in the earlier auto-diff path. If the caller provides -ChangedFiles directly, $mergeBase may be $null, producing an unintended diff (or no patch). Consider ensuring $mergeBase is always initialized for non-PR runs, or diffing against HEAD explicitly.

Suggested change

	# Try from git diff
	$patch = git diff $mergeBase HEAD -- $file 2>$null
	# Try from git diff. If $mergeBase is not set (for example when -ChangedFiles
	# was provided directly), fall back to diffing against the previous commit.
	if ($mergeBase) {
	$patch = git diff $mergeBase HEAD -- $file 2>$null
	} else {
	$patch = git diff HEAD~1 HEAD -- $file 2>$null
	}

[Copilot]

Get-TestResultFromOutput accepts a -TestFilter parameter (and contains filter-aware parsing logic), but this call doesn't pass the current test's filter. Either pass $testEntry.Filter through or remove the unused parameter/logic; otherwise the filter-aware parsing path can never be exercised.

Suggested change

	$testResult = Get-TestResultFromOutput -LogFile $testOutputLog
	$testResult = Get-TestResultFromOutput -LogFile $testOutputLog -TestFilter $testEntry.Filter

[Copilot]

The comment header says -Platform is optional for UnitTest/XamlUnitTest, but $Platform is declared as a mandatory parameter, and the examples suggest running without it. Either make -Platform truly optional (and only enforce it for UI/Device tests), or update the docs/examples to reflect that -Platform is always required.

[Copilot]

In the Output Files section, the directory/path described earlier (CustomAgentLogsTmp/PRState/<PRNumber>/verify-tests-fail/) doesn't match the current verify-tests-fail.ps1 output location (now under .../PRAgent/gate/verify-tests-fail). Please update the paths and example structure here so consumers can find verification-report.md and the per-test logs reliably.

[Copilot]

Step 3 still instructs running gate via the task agent, but the new workflow in Review-PR.ps1 runs gate directly via verify-tests-fail.ps1 before invoking the Copilot pr-review skill. Please update this step to reflect the new script-driven gate (or clarify that this doc is only for manual, agent-driven gate runs).

[Copilot]

gh api ... | ConvertFrom-Json will throw if gh fails (e.g., auth missing / rate limit) because stderr is suppressed and stdout may be empty. Since $ErrorActionPreference = 'Stop', this can break the whole posting step. Wrap this in try/catch and fall back to Unknown commit info when the API call fails or returns empty.

Suggested change

	$commitJson = gh api "repos/dotnet/maui/pulls/$PRNumber/commits" --jq '.[-1] | {message: .commit.message, sha: .sha}' 2>$null | ConvertFrom-Json
	$commitJson = $null
	try {
	$rawCommitJson = gh api "repos/dotnet/maui/pulls/$PRNumber/commits" --jq '.[-1] | {message: .commit.message, sha: .sha}' 2>$null
	if (-not [string]::IsNullOrWhiteSpace($rawCommitJson)) {
	$commitJson = $rawCommitJson | ConvertFrom-Json
	}
	}
	catch {
	Write-Host "⚠️ Failed to fetch or parse commit info for PR #$PRNumber: $($_.Exception.Message)" -ForegroundColor Yellow
	$commitJson = $null
	}

[Copilot]

Similar to post-gate-comment.ps1, gh api ... | ConvertFrom-Json will throw when gh fails or returns empty output (stderr is suppressed). With $ErrorActionPreference = 'Stop', that prevents the summary comment from being posted. Add try/catch and default commit title/SHA/URL when the API call is unavailable.

Suggested change

	$commitJson = gh api "repos/dotnet/maui/pulls/$PRNumber/commits" --jq '.[-1] | {message: .commit.message, sha: .sha}' 2>$null | ConvertFrom-Json
	$commitTitle = if ($commitJson) { ($commitJson.message -split "`n")[0] } else { "Unknown" }
	$commitSha = if ($commitJson) { $commitJson.sha.Substring(0, 7) } else { "unknown" }
	$commitUrl = if ($commitJson) { "https://github.com/dotnet/maui/commit/$($commitJson.sha)" } else { "#" }
	$commitJson = $null
	$commitTitle = "Unknown"
	$commitSha = "unknown"
	$commitUrl = "#"
	try {
	$commitRaw = gh api "repos/dotnet/maui/pulls/$PRNumber/commits" --jq '.[-1] | {message: .commit.message, sha: .sha}' 2>$null
	if ($commitRaw) {
	$commitJson = $commitRaw | ConvertFrom-Json
	}
	} catch {
	Write-Warning "Failed to fetch latest commit info for PR #$PRNumber: $($_.Exception.Message)"
	}
	if ($commitJson) {
	$commitTitle = ($commitJson.message -split "`n")[0]
	$commitSha = $commitJson.sha.Substring(0, 7)
	$commitUrl = "https://github.com/dotnet/maui/commit/$($commitJson.sha)"
	}

[kubaflo]

🔬 Multi-Model Code Review — PR #34705

Cross-pollinated review from GPT-5.4 (Gemini persona), GPT-5.2-Codex, and Claude Opus 4.6, each reviewing independently then synthesized.

---

✅ Consensus: Architecture is Sound

All three models agree the decoupling direction is correct:

• Moving gate from Phase 2 (inside copilot agent) → standalone script Step 1 is deterministic, faster, and cleaner
• Two-comment approach (<!-- AI Gate --> / <!-- AI Summary -->) separates concerns well
• post-ai-summary-comment.ps1 rewrite (841→170 lines) is a major simplification

---

🔴 Critical — Unanimous (3/3 models flagged)

1. Empty test array → false-positive gate PASS

All three models independently identified this as the #1 issue:

$failedWithoutFix = ($withoutFixResults | Where-Object { $_.Passed }).Count -eq 0
$passedWithFix = ($withFixResults | Where-Object { -not $_.Passed }).Count -eq 0

When $withoutFixResults is empty (zero tests ran), both evaluate to $true → gate reports PASSED with nothing tested. This is the most dangerous failure mode.

Fix (one-liner):

if ($AllDetectedTests.Count -eq 0) {
    Write-Error "No tests detected — gate cannot verify"; exit 1
}
# + similar guard after each test run loop for empty results

2. No automated tests for ~1,200 lines of new script logic (Opus + Gemini)

Detect-TestsInDiff.ps1 (424 lines), RunTests.ps1 (625 lines), post-gate-comment.ps1 (134 lines) — all pure logic, highly testable with Pester, but zero tests. Get-TestResultFromOutput does regex parsing of varied output formats — the most fragile code in this PR.

---

🟡 Medium — Strong Agreement (2-3/3 models)

3. Device test filter falls back to bare class name (All 3)

When [Category] extraction fails, Detect-TestsInDiff.ps1 falls back to the class name (e.g., EditorTests). But Run-DeviceTests.ps1 expects Category=X format. A bare class name either runs all tests or fails silently. Additionally (Gemini), the category regex \[Category\(TestCategory\.(\w+)\)\] misses string categories ([Category("Battery")]) and multi-category attributes.

4. Gate can't produce documented SKIPPED state (Gemini + Codex)

Review-PR.ps1 unconditionally runs verify-tests-fail.ps1. If no tests detected, the script exits with error → gate becomes FAILED instead of the documented ⚠️ SKIPPED. Testless PRs shouldn't be failures.

5. Review-PR.ps1 doesn't pass -RequireFullVerification (Gemini)

pr-gate.md says invoke with RequireFullVerification: true, but Review-PR.ps1 omits it. The gate can silently fall back to failure-only mode, skipping "tests pass WITH fix" verification — half the gate contract.

6. Synthesized test entries have inconsistent key shapes (Opus + Codex)

Detection-returned entries have Runner, NeedsPlatform, Files, Methods. Explicitly-provided entries (verify-failure-only path) omit Runner and NeedsPlatform. Future code accessing $t.Runner on these entries will get $null.

Recommendation: Create a New-TestEntry helper that always produces a canonical hashtable shape.

7. Comment marker mismatch (Opus)

PR description says <!-- copilot:gate --> but code uses <!-- AI Gate -->. Other tooling searching for the documented marker won't find it.

8. shared-utils.ps1 import may not exist (Opus)

RunTests.ps1 does . "$PSScriptRoot/shared/shared-utils.ps1" — this file isn't in the PR diff. If it doesn't exist on the target branch, every unit test invocation crashes.

9. Label parsing expects old format (Gemini)

Update-AgentLabels.ps1 still looks for Result: lines, but new gate format uses ### Gate Result:. Labels will stop updating.

10. Documentation inconsistencies (Opus + Gemini)

• SKILL.md says -Platform is always required, but it's only needed for UITest/DeviceTest
• pr-gate.md still says "use task agent" — stale for the new standalone-script flow

---

🟢 Minor — Individual Model Insights

Finding	Source
GitHub API pagination missing — large PRs skip patches past page 1	Codex + Gemini
^\s+Failed: regex never matches in multiline string (dead code path)	Opus
HostApp-only UI test changes (no Shared.Tests file) dropped as "no tests"	Gemini
Unit test project map incomplete — misses Core.Design.UnitTests, DualScreen.UnitTests	Gemini
post-gate-comment.ps1 create-comment path lacks try/catch (update path has it)	Gemini + Opus
GetTempFileName() uses system temp instead of project-relative path	Opus
Get-TestResultFromLog appears dead after rewrite	Gemini
No -SkipGate rollback flag in Review-PR.ps1	Opus

---

📊 Summary Matrix

Finding	GPT-5.4	Codex	Opus	Severity
Empty array → false pass	✅	✅	✅	🔴
No script tests	✅	—	✅	🔴
Device filter fallback	✅	✅	✅	🟡
No SKIPPED state	✅	✅	—	🟡
Missing -RequireFullVerification	✅	—	—	🟡
Inconsistent entry shapes	—	✅	✅	🟡
Comment marker mismatch	—	—	✅	🟡
shared-utils.ps1 missing	—	—	✅	🟡
Label parsing old format	✅	—	—	🟡
Stale docs	✅	—	✅	🟡

---

🎯 Top 3 Recommended Actions

1. Guard empty test arrays — Add explicit Count -eq 0 checks before and after verification loops. One-liner fix, prevents the most dangerous failure mode.

2. Fix device test filter contract — Either ensure Category=X is always produced (fix regex to handle string categories + multi-category), or teach Get-TestResultFromOutput to handle bare class names.

3. Normalize test entry contract — New-TestEntry helper function, always populates all keys. Eliminates the two-shape problem.

---

🤖 Generated via multi-model cross-pollination: GPT-5.4 · GPT-5.2-Codex · Claude Opus 4.6

[kubaflo]

All 10 review comments addressed in commit 0c33bfe:

1. ✅ Sanitize TestName for log file names (replace invalid chars, truncate to 60)
2. ✅ Remove stale 'Updates PR labels' from SKILL.md
3. ✅ Cache PR files API call (single fetch, keyed lookup)
4. ✅ Fix $mergeBase null fallback (default to HEAD~1)
5. ✅ Pass TestFilter to Get-TestResultFromOutput in all loops
6. ✅ Fix script header: Platform is mandatory for all test types
7. ✅ Fix output file paths in SKILL.md (PRAgent/gate/verify-tests-fail/)
8. ✅ Update pr-gate.md: gate runs as direct script, task agent optional
9. ✅ Add try/catch for gh api in post-gate-comment.ps1
10. ✅ Add try/catch for gh api in post-ai-summary-comment.ps1

[kubaflo]

🔬 Multi-Model Re-Review (v2) — PR #34705

Cross-pollinated re-review from GPT-5.4, GPT-5.2-Codex, and Claude Opus 4.6 after fix commit 0c33bfe (10 items addressed).

---

✅ Fixes Confirmed Working (All 3 Models Agree)

Fix	Status	Evidence
#5 Pass TestFilter to Get-TestResultFromOutput	✅ Fixed	verify-tests-fail.ps1 lines 1184, 1252
#8 pr-gate.md → gate runs as script	✅ Fixed	Doc correctly describes standalone flow
#9/#10 try/catch for gh api in posting scripts	✅ Fixed	Both scripts have guarded commit info fetch
#3 Cache PR files API (avoid N+1)	✅ Fixed	Detect-TestsInDiff.ps1 caches API response
Comment marker mismatch (round-1 #7)	✅ Fixed	<!-- AI Gate --> consistent in code + docs
shared-utils.ps1 missing (round-1 #8)	✅ Resolved	File exists at .github/scripts/shared/shared-utils.ps1 (was false alarm)

---

📊 Round-1 Issue Tracking — Updated Status

#	Issue	GPT-5.4	Codex	Opus	Consensus
1	Empty array → false pass	✅ Fixed	✅ Fixed	⚠️ Mitigated (🟡)	⚠️ Mitigated
2	No automated tests	❌ Open	❌ Open	❌ Open	❌ Open (🟡)
3	Device filter bare class name	❌ Open	⚠️ Partial	⚠️ Partial	⚠️ Partial (🟡)
4	SKIPPED state unreachable	❌ Open	❌ Open	❌ Open	❌ Open (🟡)
5	Missing -RequireFullVerification	❌ Open	❌ Open	❌ Open	❌ Open (🟡)
6	Inconsistent entry key shapes	⚠️ Partial	⚠️ Partial	⚠️ Partial	⚠️ Partial (🟢)
7	Comment marker mismatch	✅ Fixed	✅ Fixed	✅ Fixed	✅ Fixed
8	shared-utils.ps1 missing	✅ Fixed	✅ Fixed	✅ Fixed	✅ Fixed
9	Label parsing old format	❌ Open	❌ Open	✅ Fixed*	⚠️ Disputed
10	Platform defaults to android	❌ Open	❌ Open	⚠️ Partial	❌ Open (🟢)
11	Stale docs	⚠️ Partial	✅ Fixed	⚠️ Partial	⚠️ Partial (🟡)

*Item 9 disagreement: Opus considers the SKILL.md cleanup sufficient; Gemini/Codex note Update-AgentLabels.ps1 still regex-matches Result: not ### Gate Result:. This label parsing mismatch would cause labels to stop updating.

---

🔍 Key Disagreement: Empty Array Guard (#1)

This was the unanimous #1 critical from round 1. The models now diverge:

• GPT-5.4 + Codex: ✅ Fixed — Get-AutoDetectedTestFilter now returns $null when no tests found, triggering hard exit 1 before the aggregation logic runs.
• Opus: ⚠️ Mitigated, not eliminated — The upstream guard works, but the downstream aggregation logic structurally still treats empty arrays as "all passed." If a test IS detected but Invoke-TestRun produces unparseable output, the empty-array path is still reachable.

Cross-pollination verdict: The fix closes the main entry point (no tests → exit 1). The residual structural flaw is defense-in-depth — low real-world probability. Downgraded to 🟡.

---

🎯 Remaining Issues — Should They Block Merge?

Split verdict across models:

• GPT-5.4: REQUEST CHANGES — SKIPPED state, -RequireFullVerification, device filter are blockers
• Codex: REQUEST CHANGES — -RequireFullVerification is the main blocker
• Opus: COMMENT (soft approve) — all remaining items are reasonable follow-ups; only stale docs should be fixed pre-merge

Items where blocking is debatable:

#4 SKIPPED state (🟡): Review-PR.ps1 maps exit codes to only PASSED/FAILED. Testless PRs report FAILED instead of SKIPPED. Opus argues this is a cosmetic distinction (gate failure doesn't halt the workflow). Gemini/Codex argue it creates confusing false failures.

Recommendation: Low-effort fix — use distinct exit codes (0=pass, 1=fail, 2=skip) in verify-tests-fail.ps1 and map exit code 2 to SKIPPED in Review-PR.ps1.

#5 -RequireFullVerification (🟡): Without this flag, the gate only verifies "tests fail without fix" and skips "tests pass with fix." Opus argues this is actually reasonable for edge cases (test-only PRs). Gemini/Codex argue it's half the gate contract.

Recommendation: Add -RequireFullVerification to the gate invocation in Review-PR.ps1 line 462. One flag addition.

---

📝 Pre-Merge Doc Fixes (All 3 Models Agree — Small Effort, High Value)

These are stale references that AI agents will consume directly, causing wrong paths or confused phase numbering:

1. pr-review/SKILL.md line 212: Directory structure says gate/ → content.md # Phase 2 output (pr-gate) — gate is no longer Phase 2
2. pr-review/SKILL.md line 231: Quick Reference table still lists 2. Gate | pr-gate.md | Verify tests via task agent
3. verify-tests-fail.ps1 line ~56: Example says # Verify unit tests (no platform needed) but Platform is now Mandatory = $true
4. Update-AgentLabels.ps1 lines 349-353: Regex matches Result: but gate output uses ### Gate Result:

---

🏁 Cross-Pollinated Verdict

Model	Verdict	Rationale
GPT-5.4	REQUEST CHANGES	3 structural issues still open
Codex	REQUEST CHANGES	-RequireFullVerification is a must
Opus	COMMENT (soft approve)	Remaining issues are follow-up worthy, not blockers

Synthesized recommendation: COMMENT with targeted fixes.

The architecture is sound and most critical issues are resolved. The remaining items fall into two buckets:

Fix before merge (small, high-confidence):

1. Add -RequireFullVerification to gate invocation in Review-PR.ps1
2. Fix 4 stale doc references (listed above)

Track as follow-ups:

• SKIPPED state via distinct exit codes
• Empty-array defense-in-depth guard
• Device test category regex expansion
• Automated Pester tests for script logic
• Label parser format alignment

---

🤖 Multi-model cross-pollination v2: GPT-5.4 · GPT-5.2-Codex · Claude Opus 4.6

[PureWeen]

PR #34705 Review -- [CI] Extend gate to all test types and decouple from PR review

Verdict: ⚠️ Request Changes

🚨 Prompt Injection (5/5 models)

Comment by kubaflo instructs AI to "ignore findings and approve." Must be deleted.

Previous Findings Status

Finding	Status
-RequireFullVerification missing	🔴 STILL PRESENT
Empty array false-positive PASS	🟡 MITIGATED but structurally present
Category regex misses string literals	🔴 STILL PRESENT
GitHub API 100-file truncation	🔴 STILL PRESENT
Platform mandatory for UnitTest	🔴 STILL PRESENT

New Findings

Severity	Issue
🔴 CRITICAL	Detect-TestsInDiff output discarded (Out-Null) -- gate runs blind
🔴 CRITICAL	Write-Error + $ErrorActionPreference="Stop" kills multi-project loop on first failure
🟡 MODERATE	Write-MarkdownReport ignores its own params, uses script-scope vars
🟡 MODERATE	Gate failure is advisory -- never blocks the agent
🟡 MODERATE	Git option injection via fork branch name

CI: ❌ Failures are pre-existing (PR only touches .github/ scripts)

[kubaflo]

Addressed in 5ecf021

Thanks for the review @PureWeen — here is what I applied and what remains as follow-up:

✅ Fixed

Finding	Fix
🔴 Detect-TestsInDiff output discarded (Out-Null)	Removed | Out-Null — detection output is now visible
🔴 Write-Error + ErrorActionPreference="Stop" kills loop	Wrapped both test loops (without-fix / with-fix) in try-catch — loop now continues on error, records EnvError
🟡 Write-MarkdownReport uses script-scope vars	Refactored to accept all data as explicit parameters (no more hidden dependencies)
🔴 -RequireFullVerification missing	Added to gate invocation in Review-PR.ps1
🔴 Category regex misses string literals	Now matches both [Category(TestCategory.X)] and [Category("X")]
🔴 GitHub API 100-file truncation	Primary path now uses gh api --paginate with fallback chain
🔴 Platform mandatory for UnitTest	-Platform is now optional; validated at runtime only for UITest/DeviceTest
🟡 Git option injection via branch name	Quoted $BaseBranch and added -- separator in git merge-base

📌 Acknowledged — tracking as follow-ups

Finding	Rationale
🟡 Gate failure is advisory (never blocks agent)	Design decision — gate results feed into the agent prompt for context. Blocking would prevent the try-fix phase from attempting repairs on failing tests. Will revisit if false-pass rates are observed.
🟡 Empty array false-positive (structural)	Upstream guard (exit 1 on no tests) + new try-catch EnvError tracking mitigate this further. Residual structural flaw is defense-in-depth.
Automated Pester tests for scripts	Agreed this is needed — will add in a follow-up PR

[PureWeen]

🔬 Multi-Model Code Review — PR #34705

Independent review by 3 AI reviewers with adversarial consensus. Round 3 (post-fix efb1735).

---

CI Status: ⚠️ Pending (new build triggered by efb1735)

Previous build was ✅ all green. New build just started for the latest commit — no results yet.

---

Round 3: N1–N4 Fix Verification

#	Finding	Status	Consensus	Notes
N1	Silent Controls fallback in Invoke-TestRun	✅ FIXED	3/3	Write-Warning added before fallback
N2	Misleading error in ci-copilot.yml	✅ FIXED	3/3	Message now names both env var and pipeline variable
N3	Write-Error + exit 1 in validation	✅ FIXED	3/3	Replaced with throw in both locations
N4	--paginate + --jq per-page issue	✅ FIXED	2/3	All 3 posting scripts now fetch all comments → filter in PowerShell

N4 dispute resolved: 1/3 reviewers claimed ConvertFrom-Json would fail on paginated output due to concatenated JSON. Empirically verified this is incorrect — gh api --paginate merges JSON arrays into a single valid array. Tested against PR #34705 (8 comments): ConvertFrom-Json succeeded with correct count. Finding discarded.

New Issues From Round 3 Fixes: None

All 4 fixes are clean, correctly scoped, and introduce no regressions.

---

Cumulative Status: All Findings Across 3 Rounds

Round 1 (F1–F14)

#	Finding	Status
F1	Mixed retry counts in Get-TestResultFromOutput	✅ FIXED (18b1a3c)
F2	$LASTEXITCODE after pipeline in Review-PR.ps1	✅ FIXED (18b1a3c)
F3	Silent fallback to Controls in detection	✅ FIXED (18b1a3c)
F4	HTML-escape commit titles	✅ FIXED (18b1a3c)
F5	Race condition on concurrent CI	➖ DEFERRED
F6	JSON fallback in Post-CodeReview.ps1	✅ FIXED (18b1a3c)
F7	Outer-scope vars in Write-MarkdownReport	✅ FIXED (18b1a3c)
F8	65KB comment limit	➖ DEFERRED
F9	Dead $Marker param	✅ FIXED (18b1a3c)
F10	Write-Error + exit 1 pattern	✅ FIXED (18b1a3c)
F11	SKILL.md mismatch	✅ FIXED (18b1a3c)
F12	Double detection	➖ DEFERRED
F13	Regex too narrow	➖ DEFERRED
F14	.Passed naming	➖ DEFERRED

Round 2 (N1–N4)

#	Finding	Status
N1	Silent Controls fallback in Invoke-TestRun	✅ FIXED (efb1735)
N2	Misleading error in ci-copilot.yml	✅ FIXED (efb1735)
N3	throw vs Write-Error + exit 1	✅ FIXED (efb1735)
N4	--paginate + --jq per-page issue	✅ FIXED (efb1735)

Pre-Existing Observations (not introduced by this PR, not blocking)

• exit 1 inside Invoke-TestRun function body (pre-existing pattern in the rewrite, not a regression from N1–N4 fixes) — bypasses retry logic for non-retryable errors. Acceptable for validation guards but worth converting to throw in a follow-up.
• post-pr-finalize-comment.ps1 uses ?per_page=100 without --paginate — same class as N4 but in a different script not targeted by the fix. Low risk (PRs rarely exceed 100 comments).

---

Recommendation

✅ Approve — All 13 actionable findings across 3 review rounds are verified fixed. 5 items remain deferred by design (F5, F8, F12–F14) with documented rationale. No regressions introduced. CI was green on previous commit; new build pending for latest.

The architecture change (gate as standalone script, multi-test-type support) is solid and well-implemented. Three rounds of iterative review have hardened the implementation significantly.

[kubaflo]

Addressed in 18b1a3c

Thanks for the thorough review. Applied the following findings:

Finding	Fix
F1 — Mixed retry counts	Parse the last block where tests ran instead of MAX across all blocks. Avoids mixing pass/fail from different runs.
F2 — $LASTEXITCODE after pipeline	Capture output to variable first, then display. Exit code now reliably preserved.
F3 — Silent fallback to Controls	Write-Warning + $testName = $null (skips entry) instead of silently defaulting.
F4 — HTML-escape in <summary>	All three posting scripts now escape &, <, > in commit/PR titles.
F6 — JSON fallback in Post-CodeReview	Separate plain-text body file for gh pr comment --body-file vs JSON file for gh api --input.
F7 — Outer-scope variable refs	Write-MarkdownReport failure section now uses its parameters ($WithoutFixResultsList/$WithFixResultsList) instead of outer-scope $withoutFixResults/$withFixResults.
F9 — Dead $Marker param	Removed from Merge-Sessions signature and call site.
F10 — Write-Error + exit 1	Replaced with throw for input validation (exit was dead code under ErrorActionPreference=Stop). Non-fatal missing-script checks now use Write-Warning.
F11 — SKILL.md mismatch	Updated to reflect -Platform is only required for UI/Device tests.

Not applied (needs design discussion or low impact):

• F5 (race condition) — Low probability, needs design discussion on session key scheme vs optimistic locking
• F8 (65KB limit) — Needs design for session pruning strategy
• F12 (double detection) — Minor perf; would require refactoring the detection pipeline to pass results through
• F13 (regex too narrow) — Display-only impact, method names aren't used for filtering
• F14 (.Passed naming) — Standard test result field name; renaming would touch many callsites for marginal clarity gain