Skip to content

[main] Update dependencies from dotnet/xharness#35488

Open
dotnet-maestro[bot] wants to merge 4 commits into
mainfrom
darc-main-6a1e8005-f556-4791-bfe8-44f81eccde28
Open

[main] Update dependencies from dotnet/xharness#35488
dotnet-maestro[bot] wants to merge 4 commits into
mainfrom
darc-main-6a1e8005-f556-4791-bfe8-44f81eccde28

Conversation

@dotnet-maestro

@dotnet-maestro dotnet-maestro Bot commented May 18, 2026

Copy link
Copy Markdown
Contributor

This pull request updates the following dependencies

From https://github.com/dotnet/xharness

…60514.1

On relative base path root
Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26230.4 -> To Version 11.0.0-prerelease.26264.1
@github-actions

github-actions Bot commented May 18, 2026

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 35488

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 35488"

@kubaflo

kubaflo commented May 24, 2026

Copy link
Copy Markdown
Contributor

/review -b feature/refactor-copilot-yml

@MauiBot MauiBot added s/agent-review-incomplete s/agent-fix-pr-picked AI could not beat the PR fix - PR is the best among all candidates s/agent-reviewed PR was reviewed by AI agent workflow (full 4-phase review) labels May 24, 2026
kubaflo
kubaflo previously approved these changes May 25, 2026
@kubaflo

kubaflo commented May 28, 2026

Copy link
Copy Markdown
Contributor

/review -b feature/refactor-copilot-yml

@kubaflo

kubaflo commented May 30, 2026

Copy link
Copy Markdown
Contributor

/review -b feature/refactor-copilot-yml -p ios

@MauiBot

This comment has been minimized.

…60529.1

On relative base path root
Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26230.4 -> To Version 11.0.0-prerelease.26279.1
@kubaflo

kubaflo commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

/review -b feature/enhanced-reviewer

MauiBot

This comment was marked as outdated.

@kubaflo

kubaflo commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

/review -b feature/enhanced-reviewer -p android

@MauiBot MauiBot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — 1 findings

See inline comments for details.

Comment thread eng/Versions.props Outdated
<MicrosoftDotNetXHarnessTestRunnersXunitVersion>11.0.0-prerelease.26230.4</MicrosoftDotNetXHarnessTestRunnersXunitVersion>
<MicrosoftDotNetXHarnessCLIVersion>11.0.0-prerelease.26230.4</MicrosoftDotNetXHarnessCLIVersion>
<MicrosoftDotNetXHarnessTestRunnersCommonVersion>11.0.0-prerelease.26279.1</MicrosoftDotNetXHarnessTestRunnersCommonVersion>
<MicrosoftDotNetXHarnessTestRunnersXunitVersion>11.0.0-prerelease.26279.1</MicrosoftDotNetXHarnessTestRunnersXunitVersion>

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Build & MSBuild — Updating MicrosoftDotNetXHarnessTestRunnersXunitVersion here does not update the device-test runner package actually compiled into the device-test apps. The only Microsoft.DotNet.XHarness.TestRunners.Xunit PackageReferences I found are still hard-coded to 1.0.0-prerelease.21558.2 in src/TestUtils/src/DeviceTests/TestUtils.DeviceTests.csproj, src/TestUtils/src/DeviceTests.Runners/TestUtils.DeviceTests.Runners.csproj, and src/Core/tests/DeviceTests.Shared/Core.DeviceTests.Shared.csproj. If this PR depends on runner-side changes from 11.0.0-prerelease.26279.1 for Android/device-test filtering, CI will still build the apps with the old runner. Please switch those references to $(MicrosoftDotNetXHarnessTestRunnersXunitVersion) or update them explicitly.

@MauiBot MauiBot added s/agent-fix-win AI found a better alternative fix than the PR and removed s/agent-fix-pr-picked AI could not beat the PR fix - PR is the best among all candidates labels Jun 7, 2026

@MauiBot MauiBot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review Summary

@dotnet-maestro[bot] — new AI review results are available based on this last commit: 63b73b2.
Update dependencies from https://github.com/dotnet/xharness build 20260529.1 To request a fresh review after new comments or commits, comment /review rerun.

Gate Skipped Code Review In Review Confidence High Platform Android Requested For Testing

Review Sessions — click to expand
Gate — Test Before & After Fix

Gate Result: ⚠️ SKIPPED

No tests were detected in this PR.

Recommendation: Add tests to verify the fix using the write-tests-agent.


Pre-Flight — Context & Validation

Issue: unavailable — GitHub CLI authentication unavailable; no linked issue could be fetched.
PR: #35488 - Local squashed review branch (pr-review-35488)
Platforms Affected: android requested for testing; changed automation also affects CI/Copilot review and Windows device-test filtering.
Files Changed: 11 implementation, 6 test

Key Findings

  • GitHub PR metadata, comments, inline feedback, and linked issue could not be fetched because gh is unauthenticated in this environment.
  • Local diff changes Copilot PR-review automation, AI summary posting, stale-comment cleanup, device-test helper scripts, token usage aggregation, dependency metadata, and related tests/docs.
  • Gate was provided as skipped: no tests detected in this PR; gate verification was not re-run and gate/content.md was not touched.
  • Code review found a security-impacting issue where AI-authored recommendations can become formal GitHub PR review events, plus a Windows category-filter over-selection issue.

Code Review Summary

Verdict: NEEDS_CHANGES
Confidence: high
Errors: 1 | Warnings: 2 | Suggestions: 0

Key code review findings:

  • .github/scripts/post-ai-summary-comment.ps1:670 posts $postedEvent as a GitHub PR review event, allowing AI-authored text to submit APPROVE or REQUEST_CHANGES; a non-PR winner can also still allow APPROVE.
  • ⚠️ .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1:190 uses substring matching for explicit Category= filters, so Category=Button can select RadioButton.
  • ⚠️ CI status could not be verified with authenticated gh; local baseline Pester execution is blocked because Invoke-Pester is not installed.

Fix Candidates

# Source Approach Test Result Files Changed Notes
PR PR #35488 Existing PR automation/dependency update as represented by local squashed branch ⚠️ Gate skipped 22 files Original PR; no tests detected by prior gate

Code Review — Deep Analysis

Code Review — PR #35488

Independent Assessment

What this changes: Local diff substantially refactors Copilot review automation, AI summary posting, stale-comment cleanup, regression/device-test helpers, token-usage reporting, and XHarness dependency metadata.
Inferred motivation: Improve automated PR review reporting, deep UI-test feedback, and Copilot usage telemetry.

Reconciliation with PR Narrative

Author claims: PR body says this is an XHarness dependency update only.
Agreement/disagreement: Dependency metadata matches, but the local diff contains much broader CI/Copilot pipeline changes not described by the PR narrative.

Findings

❌ Error — Formal AI reviews can approve/request changes from automation

.github/scripts/post-ai-summary-comment.ps1:670 posts $postedEvent as a GitHub PR review event. This can be APPROVE or REQUEST_CHANGES based on AI-authored text. Worse, Get-AIReviewEventForRun allows APPROVE even when winner.json selected a non-PR try-fix candidate. This could satisfy approval-based branch protection for a PR the automation did not actually select. Keep the event as COMMENT; express recommendations in body/labels.

⚠️ Warning — Windows device test category filters over-select

.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1:190 treats parsed Category=... filters as substring matches. Category=Button will also select RadioButton, so Windows device-test runs can execute unrelated categories. Exact Category= filters should match exactly; only legacy bare filters should use substring behavior.

⚠️ Warning — CI is not green

GitHub CLI auth was unavailable, but public check-runs show Build Analysis failure and Windows device-test failures on the PR head. This also prevents LGTM under the skill rules.

Devil's Advocate

The formal-review behavior may be intentional for the trusted posting script, but allowing automated APPROVE when an alternate non-PR fix wins is inconsistent and high-impact. The category filter issue is concrete and covered by added tests that miss the Button/RadioButton case.

Verdict: NEEDS_CHANGES

Confidence: high
Summary: I found one security-impacting CI automation issue and one concrete test-selection bug. Inline findings were written to CustomAgentLogsTmp/PRState/35488/PRAgent/inline-findings.json; no GitHub comments were posted because gh auth is unavailable.


Fix — Analysis & Comparison

Fix Candidates

# Source Approach Test Result Files Changed Notes
1 maui-expert-reviewer Move AI summary from PR reviews to issue comments ⚠️ Blocked 3 files Strongest security isolation, but broader downstream compatibility risk and Pester unavailable
2 maui-expert-reviewer Keep PR review artifact but force submitted event to COMMENT ⚠️ Blocked 2 files Narrowest security fix; parser/static/dry-run probes passed, Pester unavailable
3 maui-expert-reviewer Exact-match explicit Category= filters; preserve bare fuzzy filters ⚠️ Blocked 2 files Direct helper probes passed; does not address formal-review security error; Pester unavailable
PR PR #35488 Current PR fix as locally checked out ⚠️ Gate skipped 22 files Original PR; gate reported no tests detected

Cross-Pollination / Learnings

Round Input Learning
1 Code-review finding: formal AI review events Expert suggested issue-comment-only posting to eliminate branch-protection risk entirely.
2 try-fix-1 blocked by missing Pester and broad compatibility risk Expert suggested a narrower event-lock approach that keeps existing PR-review artifact shape but forces COMMENT.
3 Separate category-filter warning Expert suggested mode-aware matching so explicit Category= is exact while legacy bare filters remain fuzzy.

Exhausted: Yes — meaningfully different approaches found were broad transport change, narrow event clamp, and separate category-filter fix. Further variations would be minor rearrangements of the same root-cause hypotheses.
Selected Fix: Candidate #2 if only addressing the formal-review security issue — it is the smallest compatible mitigation. Candidate #3 should be considered as an additional independent fix for the Windows filter warning. No candidate can be marked fully passing because Pester is unavailable in this environment.


try-fix-1 narrative

try-fix-1 — Advisory issue-comment summary only

Approach: Replace the AI summary PR-review posting/update path with advisory issue comments. This differs from the PR current formal-review approach by ensuring AI-authored recommendation text cannot submit GitHub review events (APPROVE / REQUEST_CHANGES) or satisfy approval-based branch protection.

Files changed:

  • .github/scripts/post-ai-summary-comment.ps1
  • .github/scripts/Review-PR.ps1
  • eng/pipelines/ci-copilot.yml

Test results: Blocked. PowerShell parser probes passed, endpoint probes confirmed the summary script no longer posts to /pulls/{PRNumber}/reviews, and stale AI-summary review ID variables were removed from the touched summary path. Full Pester regression execution was blocked because Invoke-Pester is not installed in this environment.

Failure/blocker analysis: The candidate is logically stronger than the PR fix for the formal-review security finding, but cannot be marked passing because the repository Pester test suite could not run. The candidate also requires broader downstream compatibility review because it intentionally changes output variable and artifact names from review IDs to comment IDs.

Diff:

diff --git a/.github/scripts/Review-PR.ps1 b/.github/scripts/Review-PR.ps1
index 4dcbbda429..8881dfe2e7 100644
--- a/.github/scripts/Review-PR.ps1
+++ b/.github/scripts/Review-PR.ps1
@@ -1561,7 +1561,7 @@ if ($risksData -and ($risksData.result -eq 'REVERT' -or $risksData.result -eq 'O
 # inline-stages architecture. Both phases are expensive (build the whole
 # repo, run agents on multiple candidates) and we just need STEPs 1-3 +
 # STEP 6 (post comment) to validate that detectedCategories /
-# aiSummaryReviewId output variables flow through to the new
+# aiSummaryCommentId output variables flow through to the new
 # RunDeepUITests + UpdateAISummaryComment stages. Flip $skipGateAndTryFix
 # back to $false (or delete the wrapper) once the new pipeline stages
 # are validated end-to-end.
@@ -2222,48 +2222,48 @@ if ($env:DEFER_COMMENT_TO_STAGE3 -eq 'true') {
     Write-Host "  ⏭️ Deferred to Stage 3 (DEFER_COMMENT_TO_STAGE3=true)" -ForegroundColor Gray
     Write-Host "  ℹ️  Content files saved in CopilotLogs artifact" -ForegroundColor Gray
     # Still emit a dummy output var so Stage 3 condition works
-    Write-Host "##vso[task.setvariable variable=aiSummaryReviewId;isOutput=true]DEFERRED"
+    Write-Host "##vso[task.setvariable variable=aiSummaryCommentId;isOutput=true]DEFERRED"
 } else {
 
 # Post PR review phases (pre-flight, try-fix, report)
-$aiSummaryReviewId = $null
-$aiSummaryReviewNodeId = $null
+$aiSummaryCommentId = $null
+$aiSummaryCommentNodeId = $null
 $reviewScript = Join-Path $summaryScriptsDir "post-ai-summary-comment.ps1"
 if (Test-Path $reviewScript) {
     try {
-        Write-Host "  📝 Posting PR review summary..." -ForegroundColor Cyan
+        Write-Host "  📝 Posting PR summary comment..." -ForegroundColor Cyan
         if ($DryRun) {
             $reviewOutput = & $reviewScript -PRNumber $PRNumber -DryRun
         } else {
             $reviewOutput = & $reviewScript -PRNumber $PRNumber
         }
-        # Capture review ID from script output (format: AI_SUMMARY_REVIEW_ID=<id>)
-        $idLine = $reviewOutput | Where-Object { $_ -match '^AI_SUMMARY_REVIEW_ID=' } | Select-Object -Last 1
-        $nodeLine = $reviewOutput | Where-Object { $_ -match '^AI_SUMMARY_REVIEW_NODE_ID=' } | Select-Object -Last 1
-        if ($idLine -match '^AI_SUMMARY_REVIEW_ID=(\d+)$') {
-            $aiSummaryReviewId = $Matches[1]
-            if ($nodeLine -match '^AI_SUMMARY_REVIEW_NODE_ID=(.+)$') {
-                $aiSummaryReviewNodeId = $Matches[1]
+        # Capture comment ID from script output (format: AI_SUMMARY_COMMENT_ID=<id>)
+        $idLine = $reviewOutput | Where-Object { $_ -match '^AI_SUMMARY_COMMENT_ID=' } | Select-Object -Last 1
+        $nodeLine = $reviewOutput | Where-Object { $_ -match '^AI_SUMMARY_COMMENT_NODE_ID=' } | Select-Object -Last 1
+        if ($idLine -match '^AI_SUMMARY_COMMENT_ID=(\d+)$') {
+            $aiSummaryCommentId = $Matches[1]
+            if ($nodeLine -match '^AI_SUMMARY_COMMENT_NODE_ID=(.+)$') {
+                $aiSummaryCommentNodeId = $Matches[1]
             }
-            Write-Host "  ✅ PR review summary posted (review ID: $aiSummaryReviewId)" -ForegroundColor Green
+            Write-Host "  ✅ PR summary comment posted (comment ID: $aiSummaryCommentId)" -ForegroundColor Green
 
-            # Persist review ID + PR number to a known location and emit
+            # Persist comment ID + PR number to a known location and emit
             # as an output variable so the downstream UpdateAISummaryComment
-            # stage in ci-copilot.yml can rewrite the review body once
+            # stage in ci-copilot.yml can rewrite the comment body once
             # the deep UI tests finish on the platform-pool agents.
-            $reviewIdFile = Join-Path $RepoRoot "CustomAgentLogsTmp/PRState/$PRNumber/PRAgent/ai-summary-review-id.txt"
-            New-Item -ItemType Directory -Force -Path (Split-Path -Parent $reviewIdFile) | Out-Null
-            $aiSummaryReviewId | Set-Content $reviewIdFile -Encoding UTF8
-            if (-not [string]::IsNullOrWhiteSpace($aiSummaryReviewNodeId)) {
-                $aiSummaryReviewNodeId | Set-Content (Join-Path (Split-Path -Parent $reviewIdFile) "ai-summary-review-node-id.txt") -Encoding UTF8
-                Write-Host "##vso[task.setvariable variable=aiSummaryReviewNodeId;isOutput=true]$aiSummaryReviewNodeId"
+            $commentIdFile = Join-Path $RepoRoot "CustomAgentLogsTmp/PRState/$PRNumber/PRAgent/ai-summary-comment-id.txt"
+            New-Item -ItemType Directory -Force -Path (Split-Path -Parent $commentIdFile) | Out-Null
+            $aiSummaryCommentId | Set-Content $commentIdFile -Encoding UTF8
+            if (-not [string]::IsNullOrWhiteSpace($aiSummaryCommentNodeId)) {
+                $aiSummaryCommentNodeId | Set-Content (Join-Path (Split-Path -Parent $commentIdFile) "ai-summary-comment-node-id.txt") -Encoding UTF8
+                Write-Host "##vso[task.setvariable variable=aiSummaryCommentNodeId;isOutput=true]$aiSummaryCommentNodeId"
             }
-            Write-Host "##vso[task.setvariable variable=aiSummaryReviewId;isOutput=true]$aiSummaryReviewId"
+            Write-Host "##vso[task.setvariable variable=aiSummaryCommentId;isOutput=true]$aiSummaryCommentId"
         } else {
-            Write-Host "  ✅ PR review summary posted" -ForegroundColor Green
+            Write-Host "  ✅ PR summary comment posted" -ForegroundColor Green
         }
     } catch {
-        Write-Host "  ⚠️ PR review summary posting failed (non-fatal): $_" -ForegroundColor Yellow
+        Write-Host "  ⚠️ PR summary comment posting failed (non-fatal): $_" -ForegroundColor Yellow
     }
 } else {
     Write-Host "  ⚠️ post-ai-summary-comment.ps1 not found — skipping review summary" -ForegroundColor Yellow
diff --git a/.github/scripts/post-ai-summary-comment.ps1 b/.github/scripts/post-ai-summary-comment.ps1
index cb22f4de94..01bb085f51 100644
--- a/.github/scripts/post-ai-summary-comment.ps1
+++ b/.github/scripts/post-ai-summary-comment.ps1
@@ -406,28 +406,24 @@ function Get-AIReviewEventForRun {
     return $reviewEvent
 }
 
-function Invoke-PostPullRequestReview {
+function Invoke-PostIssueComment {
     param(
         [Parameter(Mandatory = $true)]
         [int]$PRNumber,
 
         [Parameter(Mandatory = $true)]
-        [string]$Body,
-
-        [Parameter(Mandatory = $true)]
-        [ValidateSet('APPROVE', 'REQUEST_CHANGES', 'COMMENT')]
-        [string]$Event
+        [string]$Body
     )
 
     $tempFile = [System.IO.Path]::GetTempFileName()
     try {
-        @{ body = $Body; event = $Event } |
+        @{ body = $Body } |
             ConvertTo-Json -Depth 10 |
             Set-Content -Path $tempFile -Encoding UTF8
 
-        $response = gh api --method POST "repos/dotnet/maui/pulls/$PRNumber/reviews" --input $tempFile 2>&1
+        $response = gh api --method POST "repos/dotnet/maui/issues/$PRNumber/comments" --input $tempFile 2>&1
         if ($LASTEXITCODE -ne 0) {
-            throw "POST review failed (exit code $LASTEXITCODE): $response"
+            throw "POST issue comment failed (exit code $LASTEXITCODE): $response"
         }
 
         return (($response -join [Environment]::NewLine) | ConvertFrom-Json)
@@ -664,31 +660,19 @@ if (Get-Command Hide-StaleMauiBotPullRequestReviews -ErrorAction SilentlyContinu
         -DismissFormalReviews
 }
 
-Write-Host "Creating new AI Summary PR review ($reviewEvent)..." -ForegroundColor Yellow
-$postedEvent = $reviewEvent
-try {
-    $review = Invoke-PostPullRequestReview -PRNumber $PRNumber -Body $commentBody -Event $postedEvent
-} catch {
-    if ($postedEvent -eq 'COMMENT') {
-        throw
-    }
-
-    Write-Host "⚠️ Formal $postedEvent review was rejected; retrying as COMMENT: $_" -ForegroundColor Yellow
-    $postedEvent = 'COMMENT'
-    $review = Invoke-PostPullRequestReview -PRNumber $PRNumber -Body $commentBody -Event $postedEvent
-}
+Write-Host "Creating new AI Summary issue comment (advisory only)..." -ForegroundColor Yellow
+$comment = Invoke-PostIssueComment -PRNumber $PRNumber -Body $commentBody
 
-$reviewId = [string]$review.id
-$reviewNodeId = [string]$review.node_id
+$commentId = [string]$comment.id
+$commentNodeId = [string]$comment.node_id
 
-if (-not [string]::IsNullOrWhiteSpace($reviewId)) {
-    Set-Content -Path (Join-Path $PRAgentDir "ai-summary-review-id.txt") -Value $reviewId -Encoding UTF8
+if (-not [string]::IsNullOrWhiteSpace($commentId)) {
+    Set-Content -Path (Join-Path $PRAgentDir "ai-summary-comment-id.txt") -Value $commentId -Encoding UTF8
 }
-if (-not [string]::IsNullOrWhiteSpace($reviewNodeId)) {
-    Set-Content -Path (Join-Path $PRAgentDir "ai-summary-review-node-id.txt") -Value $reviewNodeId -Encoding UTF8
+if (-not [string]::IsNullOrWhiteSpace($commentNodeId)) {
+    Set-Content -Path (Join-Path $PRAgentDir "ai-summary-comment-node-id.txt") -Value $commentNodeId -Encoding UTF8
 }
 
-Write-Host "✅ AI Summary PR review posted (ID: $reviewId, event: $postedEvent)" -ForegroundColor Green
-Write-Output "AI_SUMMARY_REVIEW_ID=$reviewId"
-Write-Output "AI_SUMMARY_REVIEW_NODE_ID=$reviewNodeId"
-Write-Output "AI_SUMMARY_REVIEW_EVENT=$postedEvent"
+Write-Host "✅ AI Summary issue comment posted (ID: $commentId)" -ForegroundColor Green
+Write-Output "AI_SUMMARY_COMMENT_ID=$commentId"
+Write-Output "AI_SUMMARY_COMMENT_NODE_ID=$commentNodeId"
diff --git a/eng/pipelines/ci-copilot.yml b/eng/pipelines/ci-copilot.yml
index 0745323e92..47a100b2fb 100644
--- a/eng/pipelines/ci-copilot.yml
+++ b/eng/pipelines/ci-copilot.yml
@@ -739,7 +739,7 @@ stages:
               fi
 
               echo "Review output saved to $(Build.ArtifactStagingDirectory)/copilot-logs/"
-            name: RunPost                       # Stage 3 (UpdateAISummaryComment) reads aiSummaryReviewId via $(stageDependencies.ReviewPR.CopilotReview.outputs['RunPost.aiSummaryReviewId']). Note: detectedCategories comes from RunGate, not RunPost.
+            name: RunPost                       # Stage 3 (UpdateAISummaryComment) reads aiSummaryCommentId via $(stageDependencies.ReviewPR.CopilotReview.outputs['RunPost.aiSummaryCommentId']). Note: detectedCategories comes from RunGate, not RunPost.
             displayName: 'Task 4: Post (comments + labels)'
             env:
               GH_TOKEN: $(GH_COMMENT_TOKEN)
@@ -1322,7 +1322,7 @@ stages:
   #  STAGE: PostAISummaryComment
   # ─────────────────────────────────────────────────────────────────────────────
   # Final stage. Depends on both ReviewPR (which posted the initial AI
-  # summary review and emitted aiSummaryReviewId) and RunDeepUITests
+  # summary comment and emitted aiSummaryCommentId) and RunDeepUITests
   # (which produced the TRX artifacts on the right pool). Downloads the
   # artifacts, parses them via Aggregate-UITestArtifacts.ps1, and edits
   # the existing PR review to replace the in-process STEP 3 section
@@ -1332,16 +1332,16 @@ stages:
     dependsOn:
       - ReviewPR
       - RunDeepUITests
-    condition: and(in(dependencies.RunDeepUITests.result, 'Succeeded', 'SucceededWithIssues', 'Failed', 'Skipped'), or(ne(dependencies.ReviewPR.outputs['CopilotReview.RunPost.aiSummaryReviewId'], ''), in(dependencies.RunDeepUITests.result, 'Succeeded', 'SucceededWithIssues', 'Failed')))
+    condition: and(in(dependencies.RunDeepUITests.result, 'Succeeded', 'SucceededWithIssues', 'Failed', 'Skipped'), or(ne(dependencies.ReviewPR.outputs['CopilotReview.RunPost.aiSummaryCommentId'], ''), in(dependencies.RunDeepUITests.result, 'Succeeded', 'SucceededWithIssues', 'Failed')))
     jobs:
       - job: UpdateComment
         displayName: 'Post AI summary review with deep test results'
         # Job-level variables can use $[ stageDependencies... ] (cross-stage,
         # job context). The stage condition above already gated emptiness;
-        # this just makes the value available as $(aiSummaryReviewId)
+        # this just makes the value available as $(aiSummaryCommentId)
         # inside the steps.
         variables:
-          aiSummaryReviewId: $[ stageDependencies.ReviewPR.CopilotReview.outputs['RunPost.aiSummaryReviewId'] ]
+          aiSummaryCommentId: $[ stageDependencies.ReviewPR.CopilotReview.outputs['RunPost.aiSummaryCommentId'] ]
         pool:
           name: Azure Pipelines
           vmImage: ubuntu-22.04
@@ -1378,8 +1378,8 @@ stages:
               $artDir = "$(Pipeline.Workspace)/drop-deep-uitests"
               $copilotLogsDir = "$(Pipeline.Workspace)/CopilotLogs"
               $prNumber = "${{ parameters.PRNumber }}"
-              $reviewId = "$(aiSummaryReviewId)"
-              $isDeferred = ([string]::IsNullOrWhiteSpace($reviewId) -or $reviewId -eq 'DEFERRED')
+              $commentId = "$(aiSummaryCommentId)"
+              $isDeferred = ([string]::IsNullOrWhiteSpace($commentId) -or $commentId -eq 'DEFERRED')
 
               # Diagnostic logging for Stage 3 debugging
               Write-Host "=== Stage 3 Diagnostics ===" -ForegroundColor Cyan
@@ -1395,13 +1395,13 @@ stages:
                 }
               }
 
-              if ([string]::IsNullOrWhiteSpace($reviewId)) {
+              if ([string]::IsNullOrWhiteSpace($commentId)) {
                 # Reviewer crashed before posting the initial review. If deep
                 # tests produced results, fall back to DEFERRED mode to post
                 # a degraded review with test results only.
                 if (Test-Path $artDir) {
                   Write-Host "No AI summary review ID but deep test artifacts exist — falling back to DEFERRED mode"
-                  $reviewId = 'DEFERRED'
+                  $commentId = 'DEFERRED'
                   $isDeferred = $true
                 } else {
                   Write-Host "No AI summary review ID and no deep test artifacts — nothing to do"
@@ -1676,9 +1676,9 @@ stages:
                   Write-Host "No deep results and review already exists — nothing to patch"
                   exit 0
                 }
-                $existing = (gh api "repos/dotnet/maui/pulls/$prNumber/reviews/$reviewId" --jq '.body') -join [Environment]::NewLine

try-fix-2 narrative

try-fix-2 — Comment-only PR review event lock

Approach: Keep the existing AI summary PR-review artifact and downstream review-ID flow, but force the submitted GitHub review event to COMMENT. The parsed recommendation remains available for display/status only. This differs from try-fix-1 because it does not move the artifact to issue comments or rename downstream variables.

Files changed:

  • .github/scripts/post-ai-summary-comment.ps1
  • .github/skills/code-review/SKILL.md

Test results: Blocked. PowerShell parser probe passed. Static probe confirmed $reviewEvent = 'COMMENT' and that the recommendation event does not feed Invoke-PostPullRequestReview. Synthetic dry-runs with Final Recommendation: APPROVE and Final Recommendation: REQUEST CHANGES both printed Review event: COMMENT. Full Pester regression execution was blocked because Invoke-Pester is not installed.

Failure/blocker analysis: This is the lowest-risk candidate for the formal-review security finding because it preserves the existing artifact type and IDs while removing formal approval/request-changes capability. It cannot be marked passing without Pester, but it avoids the broader compatibility risk identified in try-fix-1.

Diff:

diff --git a/.github/scripts/post-ai-summary-comment.ps1 b/.github/scripts/post-ai-summary-comment.ps1
index cb22f4de94..a0704273c3 100644
--- a/.github/scripts/post-ai-summary-comment.ps1
+++ b/.github/scripts/post-ai-summary-comment.ps1
@@ -506,7 +506,8 @@ if (-not $gateSection -and $phaseSections.Count -eq 0) {
     throw "No gate or phase content found. Ensure at least one of gate/content.md or {phase}/content.md exists in $PRAgentDir."
 }
 
-$reviewEvent = Get-AIReviewEventForRun -ReportContent $phaseContentByKey['report'] -PRAgentDir $PRAgentDir
+$recommendationEvent = Get-AIReviewEventForRun -ReportContent $phaseContentByKey['report'] -PRAgentDir $PRAgentDir
+$reviewEvent = 'COMMENT'
 Write-Host "  🧾 PR review event: $reviewEvent" -ForegroundColor Cyan
 
 # ============================================================================
@@ -596,7 +597,7 @@ if ($prAuthor) {
     $authorPing += ' To request a fresh review after new comments or commits, comment `/review rerun`.'
 }
 
-$reviewStatus = switch ($reviewEvent) {
+$reviewStatus = switch ($recommendationEvent) {
     'APPROVE' { 'LGTM' }
     'REQUEST_CHANGES' { 'Needs Changes' }
     default { 'In Review' }
diff --git a/.github/skills/code-review/SKILL.md b/.github/skills/code-review/SKILL.md
index dc252d78a7..7cc9a8a45c 100644
--- a/.github/skills/code-review/SKILL.md
+++ b/.github/skills/code-review/SKILL.md
@@ -206,7 +206,7 @@ pwsh .github/scripts/post-inline-review.ps1 -PRNumber <PR_NUMBER>
 pwsh .github/scripts/post-ai-summary-comment.ps1

-In CI (eng/pipelines/ci-copilot.yml), Review-PR.ps1 calls both post-inline-review.ps1 (for inline findings) and post-ai-summary-comment.ps1 (for the wall-of-text from {phase}/content.md files), using GH_COMMENT_TOKEN. The trusted posting script may submit APPROVE or REQUEST_CHANGES from the final recommendation; the agent itself must not run review commands directly.
+In CI (eng/pipelines/ci-copilot.yml), Review-PR.ps1 calls both post-inline-review.ps1 (for inline findings) and post-ai-summary-comment.ps1 (for the wall-of-text from {phase}/content.md files), using GH_COMMENT_TOKEN. The trusted posting script submits the AI summary as a non-formal COMMENT review; the agent itself must not run review commands directly.



---

## try-fix-3 narrative
## try-fix-3 — Exact explicit Category filters, legacy fuzzy bare filters

**Approach:** Preserve legacy bare-filter substring matching, but treat explicit VSTest `Category=` filters as exact category names. This differs from try-fix-1 and try-fix-2 because it targets the separate device-test category over-selection finding rather than AI review event handling.

**Files changed:**
- `.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1`
- `.github/skills/run-device-tests/scripts/Run-DeviceTests.Tests.ps1`

**Test results:** Blocked. PowerShell parser probe passed. Direct helper probes passed for `Category=Button`, case-insensitive `Category=button`, multi-category `Category=Button|Category=Entry`, and legacy bare `Button` fuzzy matching. Full Pester regression execution was blocked because `Invoke-Pester` is not installed.

**Failure/blocker analysis:** The targeted functional behavior is validated by direct probes, but the candidate cannot be marked as fully passing without the repository Pester test suite. This candidate does not address the formal AI review security error, so it is not sufficient as a complete replacement for the PR fix on its own.

**Diff:**

```diff
diff --git a/.github/skills/run-device-tests/scripts/Run-DeviceTests.Tests.ps1 b/.github/skills/run-device-tests/scripts/Run-DeviceTests.Tests.ps1
index 5590ed3dee..7549c28859 100644
--- a/.github/skills/run-device-tests/scripts/Run-DeviceTests.Tests.ps1
+++ b/.github/skills/run-device-tests/scripts/Run-DeviceTests.Tests.ps1
@@ -41,6 +41,20 @@ Describe 'Windows device test category filtering' {
             Should -Be @('Window')
     }
 
+    It 'does not fuzzy-match explicit Category filters' {
+        Select-WindowsDeviceTestCategories `
+            -AllCategories @('Button', 'RadioButton', 'ImageButton', 'Entry') `
+            -Filter 'Category=Button' |
+            Should -Be @('Button')
+    }
+
+    It 'preserves fuzzy matching for legacy bare filters' {
+        Select-WindowsDeviceTestCategories `
+            -AllCategories @('Button', 'RadioButton', 'ImageButton', 'Entry') `
+            -Filter 'Button' |
+            Should -Be @('Button', 'RadioButton', 'ImageButton')
+    }
+
     It 'returns all categories when no category filter is supplied' {
         Select-WindowsDeviceTestCategories `
             -AllCategories @('Button', 'Window') `
diff --git a/.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 b/.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1
index 9d8cd1c5eb..4aaf6cfc04 100644
--- a/.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1
+++ b/.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1
@@ -183,11 +183,16 @@ function Select-WindowsDeviceTestCategories {
         return @($AllCategories)
     }
 
+    $useExactCategoryMatch = $Filter -match '(?i)\bCategory\s*='
     return @($AllCategories | Where-Object {
         $category = $_
         @($filters | Where-Object {
-            $category.Equals($_, [System.StringComparison]::OrdinalIgnoreCase) -or
-            $category.IndexOf($_, [System.StringComparison]::OrdinalIgnoreCase) -ge 0
+            if ($useExactCategoryMatch) {
+                $category.Equals($_, [System.StringComparison]::OrdinalIgnoreCase)
+            } else {
+                $category.Equals($_, [System.StringComparison]::OrdinalIgnoreCase) -or
+                $category.IndexOf($_, [System.StringComparison]::OrdinalIgnoreCase) -ge 0
+            }
         }).Count -gt 0
     })
 }

Report — Final Recommendation

Comparative Report — PR #35488

Candidates evaluated

Rank Candidate Result status Assessment
1 pr-plus-reviewer Gate skipped; sandbox feedback applied Best candidate. It preserves the PR's XHarness dependency update and fixes the expert-reviewed completeness gap by making the device-test projects consume $(MicrosoftDotNetXHarnessTestRunnersXunitVersion) instead of the old hard-coded runner version.
2 pr Gate skipped The raw PR updates XHarness CLI/test-runner metadata, but the device-test projects still compile against hard-coded Microsoft.DotNet.XHarness.TestRunners.Xunit 1.0.0-prerelease.21558.2, so the fix is likely incomplete for Android/device-test runner behavior.
3 try-fix-2 Blocked Lowest-risk STEP 5a candidate for the saved formal-review security issue because it forces AI summary PR reviews to COMMENT while preserving review-ID flow. However, it addresses a different automation issue than the current local PR diff and has blocked regression status.
4 try-fix-3 Blocked Correctly targets the saved Windows category-filter over-selection issue and includes direct helper probes, but it does not address the XHarness dependency completeness issue and has blocked regression status.
5 try-fix-1 Blocked Removes formal AI review events by moving summaries to issue comments, but it has broader downstream compatibility risk due to artifact/output variable renames and has blocked regression status.

Regression-test ranking rule

No saved candidate has a passing regression-test result. The raw PR gate was skipped because no tests were detected. try-fix-1, try-fix-2, and try-fix-3 are all explicitly marked Blocked because full Pester regression execution was unavailable. No candidate failed regression tests, so no failed candidate had to be ranked below a passing candidate.

Expert reviewer impact

The expert reviewer produced one actionable finding against eng/Versions.props:141: updating the central MicrosoftDotNetXHarnessTestRunnersXunitVersion property does not update the actual device-test runner package used by the MAUI device-test projects because those projects still hard-code an old package version. Applying that feedback creates pr-plus-reviewer, which is strictly more complete than the raw PR for the Android/device-test XHarness update.

Winner

pr-plus-reviewer is the winning candidate. It is the PR fix with targeted expert feedback applied in a sandbox copy, so it stays aligned with the submitted dependency-flow change while addressing the only high-confidence review finding. The STEP 5a try-fix candidates are useful for separate automation issues recorded in earlier logs, but they do not improve the current XHarness dependency update and are all blocked on regression execution.


Future Action — review latest findings

No alternative fix was selected for this run. Review the session findings and CI results before merging.

…60601.2

On relative base path root
Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26230.4 -> To Version 11.0.0-prerelease.26301.2
…60612.1

On relative base path root
Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26230.4 -> To Version 11.0.0-prerelease.26312.1
@kubaflo

kubaflo commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

/review rerun

@github-actions github-actions Bot added the s/agent-ready-for-rerun AI review has a new PR-author comment or commit and is ready for rerun label Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

s/agent-fix-win AI found a better alternative fix than the PR s/agent-ready-for-rerun AI review has a new PR-author comment or commit and is ready for rerun s/agent-reviewed PR was reviewed by AI agent workflow (full 4-phase review)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants