[main] Update dependencies from dotnet/xharness#35488
[main] Update dependencies from dotnet/xharness#35488dotnet-maestro[bot] wants to merge 4 commits into
Conversation
…60514.1 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26230.4 -> To Version 11.0.0-prerelease.26264.1
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 35488Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 35488" |
|
/review -b feature/refactor-copilot-yml |
|
/review -b feature/refactor-copilot-yml |
|
/review -b feature/refactor-copilot-yml -p ios |
This comment has been minimized.
This comment has been minimized.
…60529.1 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26230.4 -> To Version 11.0.0-prerelease.26279.1
|
/review -b feature/enhanced-reviewer |
|
/review -b feature/enhanced-reviewer -p android |
MauiBot
left a comment
There was a problem hiding this comment.
Expert Review — 1 findings
See inline comments for details.
| <MicrosoftDotNetXHarnessTestRunnersXunitVersion>11.0.0-prerelease.26230.4</MicrosoftDotNetXHarnessTestRunnersXunitVersion> | ||
| <MicrosoftDotNetXHarnessCLIVersion>11.0.0-prerelease.26230.4</MicrosoftDotNetXHarnessCLIVersion> | ||
| <MicrosoftDotNetXHarnessTestRunnersCommonVersion>11.0.0-prerelease.26279.1</MicrosoftDotNetXHarnessTestRunnersCommonVersion> | ||
| <MicrosoftDotNetXHarnessTestRunnersXunitVersion>11.0.0-prerelease.26279.1</MicrosoftDotNetXHarnessTestRunnersXunitVersion> |
There was a problem hiding this comment.
[major] Build & MSBuild — Updating MicrosoftDotNetXHarnessTestRunnersXunitVersion here does not update the device-test runner package actually compiled into the device-test apps. The only Microsoft.DotNet.XHarness.TestRunners.Xunit PackageReferences I found are still hard-coded to 1.0.0-prerelease.21558.2 in src/TestUtils/src/DeviceTests/TestUtils.DeviceTests.csproj, src/TestUtils/src/DeviceTests.Runners/TestUtils.DeviceTests.Runners.csproj, and src/Core/tests/DeviceTests.Shared/Core.DeviceTests.Shared.csproj. If this PR depends on runner-side changes from 11.0.0-prerelease.26279.1 for Android/device-test filtering, CI will still build the apps with the old runner. Please switch those references to $(MicrosoftDotNetXHarnessTestRunnersXunitVersion) or update them explicitly.
MauiBot
left a comment
There was a problem hiding this comment.
AI Review Summary
@dotnet-maestro[bot] — new AI review results are available based on this last commit:
63b73b2.
Update dependencies from https://github.com/dotnet/xharness build 20260529.1 To request a fresh review after new comments or commits, comment/review rerun.
Review Sessions — click to expand
Gate — Test Before & After Fix
Gate Result: ⚠️ SKIPPED
No tests were detected in this PR.
Recommendation: Add tests to verify the fix using the write-tests-agent.
Pre-Flight — Context & Validation
Issue: unavailable — GitHub CLI authentication unavailable; no linked issue could be fetched.
PR: #35488 - Local squashed review branch (pr-review-35488)
Platforms Affected: android requested for testing; changed automation also affects CI/Copilot review and Windows device-test filtering.
Files Changed: 11 implementation, 6 test
Key Findings
- GitHub PR metadata, comments, inline feedback, and linked issue could not be fetched because
ghis unauthenticated in this environment. - Local diff changes Copilot PR-review automation, AI summary posting, stale-comment cleanup, device-test helper scripts, token usage aggregation, dependency metadata, and related tests/docs.
- Gate was provided as skipped: no tests detected in this PR; gate verification was not re-run and
gate/content.mdwas not touched. - Code review found a security-impacting issue where AI-authored recommendations can become formal GitHub PR review events, plus a Windows category-filter over-selection issue.
Code Review Summary
Verdict: NEEDS_CHANGES
Confidence: high
Errors: 1 | Warnings: 2 | Suggestions: 0
Key code review findings:
- ❌
.github/scripts/post-ai-summary-comment.ps1:670posts$postedEventas a GitHub PR review event, allowing AI-authored text to submitAPPROVEorREQUEST_CHANGES; a non-PR winner can also still allowAPPROVE. ⚠️ .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1:190uses substring matching for explicitCategory=filters, soCategory=Buttoncan selectRadioButton.⚠️ CI status could not be verified with authenticatedgh; local baseline Pester execution is blocked becauseInvoke-Pesteris not installed.
Fix Candidates
| # | Source | Approach | Test Result | Files Changed | Notes |
|---|---|---|---|---|---|
| PR | PR #35488 | Existing PR automation/dependency update as represented by local squashed branch | 22 files | Original PR; no tests detected by prior gate |
Code Review — Deep Analysis
Code Review — PR #35488
Independent Assessment
What this changes: Local diff substantially refactors Copilot review automation, AI summary posting, stale-comment cleanup, regression/device-test helpers, token-usage reporting, and XHarness dependency metadata.
Inferred motivation: Improve automated PR review reporting, deep UI-test feedback, and Copilot usage telemetry.
Reconciliation with PR Narrative
Author claims: PR body says this is an XHarness dependency update only.
Agreement/disagreement: Dependency metadata matches, but the local diff contains much broader CI/Copilot pipeline changes not described by the PR narrative.
Findings
❌ Error — Formal AI reviews can approve/request changes from automation
.github/scripts/post-ai-summary-comment.ps1:670 posts $postedEvent as a GitHub PR review event. This can be APPROVE or REQUEST_CHANGES based on AI-authored text. Worse, Get-AIReviewEventForRun allows APPROVE even when winner.json selected a non-PR try-fix candidate. This could satisfy approval-based branch protection for a PR the automation did not actually select. Keep the event as COMMENT; express recommendations in body/labels.
⚠️ Warning — Windows device test category filters over-select
.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1:190 treats parsed Category=... filters as substring matches. Category=Button will also select RadioButton, so Windows device-test runs can execute unrelated categories. Exact Category= filters should match exactly; only legacy bare filters should use substring behavior.
⚠️ Warning — CI is not green
GitHub CLI auth was unavailable, but public check-runs show Build Analysis failure and Windows device-test failures on the PR head. This also prevents LGTM under the skill rules.
Devil's Advocate
The formal-review behavior may be intentional for the trusted posting script, but allowing automated APPROVE when an alternate non-PR fix wins is inconsistent and high-impact. The category filter issue is concrete and covered by added tests that miss the Button/RadioButton case.
Verdict: NEEDS_CHANGES
Confidence: high
Summary: I found one security-impacting CI automation issue and one concrete test-selection bug. Inline findings were written to CustomAgentLogsTmp/PRState/35488/PRAgent/inline-findings.json; no GitHub comments were posted because gh auth is unavailable.
Fix — Analysis & Comparison
Fix Candidates
| # | Source | Approach | Test Result | Files Changed | Notes |
|---|---|---|---|---|---|
| 1 | maui-expert-reviewer | Move AI summary from PR reviews to issue comments | 3 files | Strongest security isolation, but broader downstream compatibility risk and Pester unavailable | |
| 2 | maui-expert-reviewer | Keep PR review artifact but force submitted event to COMMENT |
2 files | Narrowest security fix; parser/static/dry-run probes passed, Pester unavailable | |
| 3 | maui-expert-reviewer | Exact-match explicit Category= filters; preserve bare fuzzy filters |
2 files | Direct helper probes passed; does not address formal-review security error; Pester unavailable | |
| PR | PR #35488 | Current PR fix as locally checked out | 22 files | Original PR; gate reported no tests detected |
Cross-Pollination / Learnings
| Round | Input | Learning |
|---|---|---|
| 1 | Code-review finding: formal AI review events | Expert suggested issue-comment-only posting to eliminate branch-protection risk entirely. |
| 2 | try-fix-1 blocked by missing Pester and broad compatibility risk | Expert suggested a narrower event-lock approach that keeps existing PR-review artifact shape but forces COMMENT. |
| 3 | Separate category-filter warning | Expert suggested mode-aware matching so explicit Category= is exact while legacy bare filters remain fuzzy. |
Exhausted: Yes — meaningfully different approaches found were broad transport change, narrow event clamp, and separate category-filter fix. Further variations would be minor rearrangements of the same root-cause hypotheses.
Selected Fix: Candidate #2 if only addressing the formal-review security issue — it is the smallest compatible mitigation. Candidate #3 should be considered as an additional independent fix for the Windows filter warning. No candidate can be marked fully passing because Pester is unavailable in this environment.
try-fix-1 narrative
try-fix-1 — Advisory issue-comment summary only
Approach: Replace the AI summary PR-review posting/update path with advisory issue comments. This differs from the PR current formal-review approach by ensuring AI-authored recommendation text cannot submit GitHub review events (APPROVE / REQUEST_CHANGES) or satisfy approval-based branch protection.
Files changed:
.github/scripts/post-ai-summary-comment.ps1.github/scripts/Review-PR.ps1eng/pipelines/ci-copilot.yml
Test results: Blocked. PowerShell parser probes passed, endpoint probes confirmed the summary script no longer posts to /pulls/{PRNumber}/reviews, and stale AI-summary review ID variables were removed from the touched summary path. Full Pester regression execution was blocked because Invoke-Pester is not installed in this environment.
Failure/blocker analysis: The candidate is logically stronger than the PR fix for the formal-review security finding, but cannot be marked passing because the repository Pester test suite could not run. The candidate also requires broader downstream compatibility review because it intentionally changes output variable and artifact names from review IDs to comment IDs.
Diff:
diff --git a/.github/scripts/Review-PR.ps1 b/.github/scripts/Review-PR.ps1
index 4dcbbda429..8881dfe2e7 100644
--- a/.github/scripts/Review-PR.ps1
+++ b/.github/scripts/Review-PR.ps1
@@ -1561,7 +1561,7 @@ if ($risksData -and ($risksData.result -eq 'REVERT' -or $risksData.result -eq 'O
# inline-stages architecture. Both phases are expensive (build the whole
# repo, run agents on multiple candidates) and we just need STEPs 1-3 +
# STEP 6 (post comment) to validate that detectedCategories /
-# aiSummaryReviewId output variables flow through to the new
+# aiSummaryCommentId output variables flow through to the new
# RunDeepUITests + UpdateAISummaryComment stages. Flip $skipGateAndTryFix
# back to $false (or delete the wrapper) once the new pipeline stages
# are validated end-to-end.
@@ -2222,48 +2222,48 @@ if ($env:DEFER_COMMENT_TO_STAGE3 -eq 'true') {
Write-Host " ⏭️ Deferred to Stage 3 (DEFER_COMMENT_TO_STAGE3=true)" -ForegroundColor Gray
Write-Host " ℹ️ Content files saved in CopilotLogs artifact" -ForegroundColor Gray
# Still emit a dummy output var so Stage 3 condition works
- Write-Host "##vso[task.setvariable variable=aiSummaryReviewId;isOutput=true]DEFERRED"
+ Write-Host "##vso[task.setvariable variable=aiSummaryCommentId;isOutput=true]DEFERRED"
} else {
# Post PR review phases (pre-flight, try-fix, report)
-$aiSummaryReviewId = $null
-$aiSummaryReviewNodeId = $null
+$aiSummaryCommentId = $null
+$aiSummaryCommentNodeId = $null
$reviewScript = Join-Path $summaryScriptsDir "post-ai-summary-comment.ps1"
if (Test-Path $reviewScript) {
try {
- Write-Host " 📝 Posting PR review summary..." -ForegroundColor Cyan
+ Write-Host " 📝 Posting PR summary comment..." -ForegroundColor Cyan
if ($DryRun) {
$reviewOutput = & $reviewScript -PRNumber $PRNumber -DryRun
} else {
$reviewOutput = & $reviewScript -PRNumber $PRNumber
}
- # Capture review ID from script output (format: AI_SUMMARY_REVIEW_ID=<id>)
- $idLine = $reviewOutput | Where-Object { $_ -match '^AI_SUMMARY_REVIEW_ID=' } | Select-Object -Last 1
- $nodeLine = $reviewOutput | Where-Object { $_ -match '^AI_SUMMARY_REVIEW_NODE_ID=' } | Select-Object -Last 1
- if ($idLine -match '^AI_SUMMARY_REVIEW_ID=(\d+)$') {
- $aiSummaryReviewId = $Matches[1]
- if ($nodeLine -match '^AI_SUMMARY_REVIEW_NODE_ID=(.+)$') {
- $aiSummaryReviewNodeId = $Matches[1]
+ # Capture comment ID from script output (format: AI_SUMMARY_COMMENT_ID=<id>)
+ $idLine = $reviewOutput | Where-Object { $_ -match '^AI_SUMMARY_COMMENT_ID=' } | Select-Object -Last 1
+ $nodeLine = $reviewOutput | Where-Object { $_ -match '^AI_SUMMARY_COMMENT_NODE_ID=' } | Select-Object -Last 1
+ if ($idLine -match '^AI_SUMMARY_COMMENT_ID=(\d+)$') {
+ $aiSummaryCommentId = $Matches[1]
+ if ($nodeLine -match '^AI_SUMMARY_COMMENT_NODE_ID=(.+)$') {
+ $aiSummaryCommentNodeId = $Matches[1]
}
- Write-Host " ✅ PR review summary posted (review ID: $aiSummaryReviewId)" -ForegroundColor Green
+ Write-Host " ✅ PR summary comment posted (comment ID: $aiSummaryCommentId)" -ForegroundColor Green
- # Persist review ID + PR number to a known location and emit
+ # Persist comment ID + PR number to a known location and emit
# as an output variable so the downstream UpdateAISummaryComment
- # stage in ci-copilot.yml can rewrite the review body once
+ # stage in ci-copilot.yml can rewrite the comment body once
# the deep UI tests finish on the platform-pool agents.
- $reviewIdFile = Join-Path $RepoRoot "CustomAgentLogsTmp/PRState/$PRNumber/PRAgent/ai-summary-review-id.txt"
- New-Item -ItemType Directory -Force -Path (Split-Path -Parent $reviewIdFile) | Out-Null
- $aiSummaryReviewId | Set-Content $reviewIdFile -Encoding UTF8
- if (-not [string]::IsNullOrWhiteSpace($aiSummaryReviewNodeId)) {
- $aiSummaryReviewNodeId | Set-Content (Join-Path (Split-Path -Parent $reviewIdFile) "ai-summary-review-node-id.txt") -Encoding UTF8
- Write-Host "##vso[task.setvariable variable=aiSummaryReviewNodeId;isOutput=true]$aiSummaryReviewNodeId"
+ $commentIdFile = Join-Path $RepoRoot "CustomAgentLogsTmp/PRState/$PRNumber/PRAgent/ai-summary-comment-id.txt"
+ New-Item -ItemType Directory -Force -Path (Split-Path -Parent $commentIdFile) | Out-Null
+ $aiSummaryCommentId | Set-Content $commentIdFile -Encoding UTF8
+ if (-not [string]::IsNullOrWhiteSpace($aiSummaryCommentNodeId)) {
+ $aiSummaryCommentNodeId | Set-Content (Join-Path (Split-Path -Parent $commentIdFile) "ai-summary-comment-node-id.txt") -Encoding UTF8
+ Write-Host "##vso[task.setvariable variable=aiSummaryCommentNodeId;isOutput=true]$aiSummaryCommentNodeId"
}
- Write-Host "##vso[task.setvariable variable=aiSummaryReviewId;isOutput=true]$aiSummaryReviewId"
+ Write-Host "##vso[task.setvariable variable=aiSummaryCommentId;isOutput=true]$aiSummaryCommentId"
} else {
- Write-Host " ✅ PR review summary posted" -ForegroundColor Green
+ Write-Host " ✅ PR summary comment posted" -ForegroundColor Green
}
} catch {
- Write-Host " ⚠️ PR review summary posting failed (non-fatal): $_" -ForegroundColor Yellow
+ Write-Host " ⚠️ PR summary comment posting failed (non-fatal): $_" -ForegroundColor Yellow
}
} else {
Write-Host " ⚠️ post-ai-summary-comment.ps1 not found — skipping review summary" -ForegroundColor Yellow
diff --git a/.github/scripts/post-ai-summary-comment.ps1 b/.github/scripts/post-ai-summary-comment.ps1
index cb22f4de94..01bb085f51 100644
--- a/.github/scripts/post-ai-summary-comment.ps1
+++ b/.github/scripts/post-ai-summary-comment.ps1
@@ -406,28 +406,24 @@ function Get-AIReviewEventForRun {
return $reviewEvent
}
-function Invoke-PostPullRequestReview {
+function Invoke-PostIssueComment {
param(
[Parameter(Mandatory = $true)]
[int]$PRNumber,
[Parameter(Mandatory = $true)]
- [string]$Body,
-
- [Parameter(Mandatory = $true)]
- [ValidateSet('APPROVE', 'REQUEST_CHANGES', 'COMMENT')]
- [string]$Event
+ [string]$Body
)
$tempFile = [System.IO.Path]::GetTempFileName()
try {
- @{ body = $Body; event = $Event } |
+ @{ body = $Body } |
ConvertTo-Json -Depth 10 |
Set-Content -Path $tempFile -Encoding UTF8
- $response = gh api --method POST "repos/dotnet/maui/pulls/$PRNumber/reviews" --input $tempFile 2>&1
+ $response = gh api --method POST "repos/dotnet/maui/issues/$PRNumber/comments" --input $tempFile 2>&1
if ($LASTEXITCODE -ne 0) {
- throw "POST review failed (exit code $LASTEXITCODE): $response"
+ throw "POST issue comment failed (exit code $LASTEXITCODE): $response"
}
return (($response -join [Environment]::NewLine) | ConvertFrom-Json)
@@ -664,31 +660,19 @@ if (Get-Command Hide-StaleMauiBotPullRequestReviews -ErrorAction SilentlyContinu
-DismissFormalReviews
}
-Write-Host "Creating new AI Summary PR review ($reviewEvent)..." -ForegroundColor Yellow
-$postedEvent = $reviewEvent
-try {
- $review = Invoke-PostPullRequestReview -PRNumber $PRNumber -Body $commentBody -Event $postedEvent
-} catch {
- if ($postedEvent -eq 'COMMENT') {
- throw
- }
-
- Write-Host "⚠️ Formal $postedEvent review was rejected; retrying as COMMENT: $_" -ForegroundColor Yellow
- $postedEvent = 'COMMENT'
- $review = Invoke-PostPullRequestReview -PRNumber $PRNumber -Body $commentBody -Event $postedEvent
-}
+Write-Host "Creating new AI Summary issue comment (advisory only)..." -ForegroundColor Yellow
+$comment = Invoke-PostIssueComment -PRNumber $PRNumber -Body $commentBody
-$reviewId = [string]$review.id
-$reviewNodeId = [string]$review.node_id
+$commentId = [string]$comment.id
+$commentNodeId = [string]$comment.node_id
-if (-not [string]::IsNullOrWhiteSpace($reviewId)) {
- Set-Content -Path (Join-Path $PRAgentDir "ai-summary-review-id.txt") -Value $reviewId -Encoding UTF8
+if (-not [string]::IsNullOrWhiteSpace($commentId)) {
+ Set-Content -Path (Join-Path $PRAgentDir "ai-summary-comment-id.txt") -Value $commentId -Encoding UTF8
}
-if (-not [string]::IsNullOrWhiteSpace($reviewNodeId)) {
- Set-Content -Path (Join-Path $PRAgentDir "ai-summary-review-node-id.txt") -Value $reviewNodeId -Encoding UTF8
+if (-not [string]::IsNullOrWhiteSpace($commentNodeId)) {
+ Set-Content -Path (Join-Path $PRAgentDir "ai-summary-comment-node-id.txt") -Value $commentNodeId -Encoding UTF8
}
-Write-Host "✅ AI Summary PR review posted (ID: $reviewId, event: $postedEvent)" -ForegroundColor Green
-Write-Output "AI_SUMMARY_REVIEW_ID=$reviewId"
-Write-Output "AI_SUMMARY_REVIEW_NODE_ID=$reviewNodeId"
-Write-Output "AI_SUMMARY_REVIEW_EVENT=$postedEvent"
+Write-Host "✅ AI Summary issue comment posted (ID: $commentId)" -ForegroundColor Green
+Write-Output "AI_SUMMARY_COMMENT_ID=$commentId"
+Write-Output "AI_SUMMARY_COMMENT_NODE_ID=$commentNodeId"
diff --git a/eng/pipelines/ci-copilot.yml b/eng/pipelines/ci-copilot.yml
index 0745323e92..47a100b2fb 100644
--- a/eng/pipelines/ci-copilot.yml
+++ b/eng/pipelines/ci-copilot.yml
@@ -739,7 +739,7 @@ stages:
fi
echo "Review output saved to $(Build.ArtifactStagingDirectory)/copilot-logs/"
- name: RunPost # Stage 3 (UpdateAISummaryComment) reads aiSummaryReviewId via $(stageDependencies.ReviewPR.CopilotReview.outputs['RunPost.aiSummaryReviewId']). Note: detectedCategories comes from RunGate, not RunPost.
+ name: RunPost # Stage 3 (UpdateAISummaryComment) reads aiSummaryCommentId via $(stageDependencies.ReviewPR.CopilotReview.outputs['RunPost.aiSummaryCommentId']). Note: detectedCategories comes from RunGate, not RunPost.
displayName: 'Task 4: Post (comments + labels)'
env:
GH_TOKEN: $(GH_COMMENT_TOKEN)
@@ -1322,7 +1322,7 @@ stages:
# STAGE: PostAISummaryComment
# ─────────────────────────────────────────────────────────────────────────────
# Final stage. Depends on both ReviewPR (which posted the initial AI
- # summary review and emitted aiSummaryReviewId) and RunDeepUITests
+ # summary comment and emitted aiSummaryCommentId) and RunDeepUITests
# (which produced the TRX artifacts on the right pool). Downloads the
# artifacts, parses them via Aggregate-UITestArtifacts.ps1, and edits
# the existing PR review to replace the in-process STEP 3 section
@@ -1332,16 +1332,16 @@ stages:
dependsOn:
- ReviewPR
- RunDeepUITests
- condition: and(in(dependencies.RunDeepUITests.result, 'Succeeded', 'SucceededWithIssues', 'Failed', 'Skipped'), or(ne(dependencies.ReviewPR.outputs['CopilotReview.RunPost.aiSummaryReviewId'], ''), in(dependencies.RunDeepUITests.result, 'Succeeded', 'SucceededWithIssues', 'Failed')))
+ condition: and(in(dependencies.RunDeepUITests.result, 'Succeeded', 'SucceededWithIssues', 'Failed', 'Skipped'), or(ne(dependencies.ReviewPR.outputs['CopilotReview.RunPost.aiSummaryCommentId'], ''), in(dependencies.RunDeepUITests.result, 'Succeeded', 'SucceededWithIssues', 'Failed')))
jobs:
- job: UpdateComment
displayName: 'Post AI summary review with deep test results'
# Job-level variables can use $[ stageDependencies... ] (cross-stage,
# job context). The stage condition above already gated emptiness;
- # this just makes the value available as $(aiSummaryReviewId)
+ # this just makes the value available as $(aiSummaryCommentId)
# inside the steps.
variables:
- aiSummaryReviewId: $[ stageDependencies.ReviewPR.CopilotReview.outputs['RunPost.aiSummaryReviewId'] ]
+ aiSummaryCommentId: $[ stageDependencies.ReviewPR.CopilotReview.outputs['RunPost.aiSummaryCommentId'] ]
pool:
name: Azure Pipelines
vmImage: ubuntu-22.04
@@ -1378,8 +1378,8 @@ stages:
$artDir = "$(Pipeline.Workspace)/drop-deep-uitests"
$copilotLogsDir = "$(Pipeline.Workspace)/CopilotLogs"
$prNumber = "${{ parameters.PRNumber }}"
- $reviewId = "$(aiSummaryReviewId)"
- $isDeferred = ([string]::IsNullOrWhiteSpace($reviewId) -or $reviewId -eq 'DEFERRED')
+ $commentId = "$(aiSummaryCommentId)"
+ $isDeferred = ([string]::IsNullOrWhiteSpace($commentId) -or $commentId -eq 'DEFERRED')
# Diagnostic logging for Stage 3 debugging
Write-Host "=== Stage 3 Diagnostics ===" -ForegroundColor Cyan
@@ -1395,13 +1395,13 @@ stages:
}
}
- if ([string]::IsNullOrWhiteSpace($reviewId)) {
+ if ([string]::IsNullOrWhiteSpace($commentId)) {
# Reviewer crashed before posting the initial review. If deep
# tests produced results, fall back to DEFERRED mode to post
# a degraded review with test results only.
if (Test-Path $artDir) {
Write-Host "No AI summary review ID but deep test artifacts exist — falling back to DEFERRED mode"
- $reviewId = 'DEFERRED'
+ $commentId = 'DEFERRED'
$isDeferred = $true
} else {
Write-Host "No AI summary review ID and no deep test artifacts — nothing to do"
@@ -1676,9 +1676,9 @@ stages:
Write-Host "No deep results and review already exists — nothing to patch"
exit 0
}
- $existing = (gh api "repos/dotnet/maui/pulls/$prNumber/reviews/$reviewId" --jq '.body') -join [Environment]::NewLinetry-fix-2 narrative
try-fix-2 — Comment-only PR review event lock
Approach: Keep the existing AI summary PR-review artifact and downstream review-ID flow, but force the submitted GitHub review event to COMMENT. The parsed recommendation remains available for display/status only. This differs from try-fix-1 because it does not move the artifact to issue comments or rename downstream variables.
Files changed:
.github/scripts/post-ai-summary-comment.ps1.github/skills/code-review/SKILL.md
Test results: Blocked. PowerShell parser probe passed. Static probe confirmed $reviewEvent = 'COMMENT' and that the recommendation event does not feed Invoke-PostPullRequestReview. Synthetic dry-runs with Final Recommendation: APPROVE and Final Recommendation: REQUEST CHANGES both printed Review event: COMMENT. Full Pester regression execution was blocked because Invoke-Pester is not installed.
Failure/blocker analysis: This is the lowest-risk candidate for the formal-review security finding because it preserves the existing artifact type and IDs while removing formal approval/request-changes capability. It cannot be marked passing without Pester, but it avoids the broader compatibility risk identified in try-fix-1.
Diff:
diff --git a/.github/scripts/post-ai-summary-comment.ps1 b/.github/scripts/post-ai-summary-comment.ps1
index cb22f4de94..a0704273c3 100644
--- a/.github/scripts/post-ai-summary-comment.ps1
+++ b/.github/scripts/post-ai-summary-comment.ps1
@@ -506,7 +506,8 @@ if (-not $gateSection -and $phaseSections.Count -eq 0) {
throw "No gate or phase content found. Ensure at least one of gate/content.md or {phase}/content.md exists in $PRAgentDir."
}
-$reviewEvent = Get-AIReviewEventForRun -ReportContent $phaseContentByKey['report'] -PRAgentDir $PRAgentDir
+$recommendationEvent = Get-AIReviewEventForRun -ReportContent $phaseContentByKey['report'] -PRAgentDir $PRAgentDir
+$reviewEvent = 'COMMENT'
Write-Host " 🧾 PR review event: $reviewEvent" -ForegroundColor Cyan
# ============================================================================
@@ -596,7 +597,7 @@ if ($prAuthor) {
$authorPing += ' To request a fresh review after new comments or commits, comment `/review rerun`.'
}
-$reviewStatus = switch ($reviewEvent) {
+$reviewStatus = switch ($recommendationEvent) {
'APPROVE' { 'LGTM' }
'REQUEST_CHANGES' { 'Needs Changes' }
default { 'In Review' }
diff --git a/.github/skills/code-review/SKILL.md b/.github/skills/code-review/SKILL.md
index dc252d78a7..7cc9a8a45c 100644
--- a/.github/skills/code-review/SKILL.md
+++ b/.github/skills/code-review/SKILL.md
@@ -206,7 +206,7 @@ pwsh .github/scripts/post-inline-review.ps1 -PRNumber <PR_NUMBER>
pwsh .github/scripts/post-ai-summary-comment.ps1-In CI (eng/pipelines/ci-copilot.yml), Review-PR.ps1 calls both post-inline-review.ps1 (for inline findings) and post-ai-summary-comment.ps1 (for the wall-of-text from {phase}/content.md files), using GH_COMMENT_TOKEN. The trusted posting script may submit APPROVE or REQUEST_CHANGES from the final recommendation; the agent itself must not run review commands directly.
+In CI (eng/pipelines/ci-copilot.yml), Review-PR.ps1 calls both post-inline-review.ps1 (for inline findings) and post-ai-summary-comment.ps1 (for the wall-of-text from {phase}/content.md files), using GH_COMMENT_TOKEN. The trusted posting script submits the AI summary as a non-formal COMMENT review; the agent itself must not run review commands directly.
---
## try-fix-3 narrative
## try-fix-3 — Exact explicit Category filters, legacy fuzzy bare filters
**Approach:** Preserve legacy bare-filter substring matching, but treat explicit VSTest `Category=` filters as exact category names. This differs from try-fix-1 and try-fix-2 because it targets the separate device-test category over-selection finding rather than AI review event handling.
**Files changed:**
- `.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1`
- `.github/skills/run-device-tests/scripts/Run-DeviceTests.Tests.ps1`
**Test results:** Blocked. PowerShell parser probe passed. Direct helper probes passed for `Category=Button`, case-insensitive `Category=button`, multi-category `Category=Button|Category=Entry`, and legacy bare `Button` fuzzy matching. Full Pester regression execution was blocked because `Invoke-Pester` is not installed.
**Failure/blocker analysis:** The targeted functional behavior is validated by direct probes, but the candidate cannot be marked as fully passing without the repository Pester test suite. This candidate does not address the formal AI review security error, so it is not sufficient as a complete replacement for the PR fix on its own.
**Diff:**
```diff
diff --git a/.github/skills/run-device-tests/scripts/Run-DeviceTests.Tests.ps1 b/.github/skills/run-device-tests/scripts/Run-DeviceTests.Tests.ps1
index 5590ed3dee..7549c28859 100644
--- a/.github/skills/run-device-tests/scripts/Run-DeviceTests.Tests.ps1
+++ b/.github/skills/run-device-tests/scripts/Run-DeviceTests.Tests.ps1
@@ -41,6 +41,20 @@ Describe 'Windows device test category filtering' {
Should -Be @('Window')
}
+ It 'does not fuzzy-match explicit Category filters' {
+ Select-WindowsDeviceTestCategories `
+ -AllCategories @('Button', 'RadioButton', 'ImageButton', 'Entry') `
+ -Filter 'Category=Button' |
+ Should -Be @('Button')
+ }
+
+ It 'preserves fuzzy matching for legacy bare filters' {
+ Select-WindowsDeviceTestCategories `
+ -AllCategories @('Button', 'RadioButton', 'ImageButton', 'Entry') `
+ -Filter 'Button' |
+ Should -Be @('Button', 'RadioButton', 'ImageButton')
+ }
+
It 'returns all categories when no category filter is supplied' {
Select-WindowsDeviceTestCategories `
-AllCategories @('Button', 'Window') `
diff --git a/.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 b/.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1
index 9d8cd1c5eb..4aaf6cfc04 100644
--- a/.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1
+++ b/.github/skills/run-device-tests/scripts/Run-DeviceTests.ps1
@@ -183,11 +183,16 @@ function Select-WindowsDeviceTestCategories {
return @($AllCategories)
}
+ $useExactCategoryMatch = $Filter -match '(?i)\bCategory\s*='
return @($AllCategories | Where-Object {
$category = $_
@($filters | Where-Object {
- $category.Equals($_, [System.StringComparison]::OrdinalIgnoreCase) -or
- $category.IndexOf($_, [System.StringComparison]::OrdinalIgnoreCase) -ge 0
+ if ($useExactCategoryMatch) {
+ $category.Equals($_, [System.StringComparison]::OrdinalIgnoreCase)
+ } else {
+ $category.Equals($_, [System.StringComparison]::OrdinalIgnoreCase) -or
+ $category.IndexOf($_, [System.StringComparison]::OrdinalIgnoreCase) -ge 0
+ }
}).Count -gt 0
})
}
Report — Final Recommendation
Comparative Report — PR #35488
Candidates evaluated
| Rank | Candidate | Result status | Assessment |
|---|---|---|---|
| 1 | pr-plus-reviewer |
Gate skipped; sandbox feedback applied | Best candidate. It preserves the PR's XHarness dependency update and fixes the expert-reviewed completeness gap by making the device-test projects consume $(MicrosoftDotNetXHarnessTestRunnersXunitVersion) instead of the old hard-coded runner version. |
| 2 | pr |
Gate skipped | The raw PR updates XHarness CLI/test-runner metadata, but the device-test projects still compile against hard-coded Microsoft.DotNet.XHarness.TestRunners.Xunit 1.0.0-prerelease.21558.2, so the fix is likely incomplete for Android/device-test runner behavior. |
| 3 | try-fix-2 |
Blocked | Lowest-risk STEP 5a candidate for the saved formal-review security issue because it forces AI summary PR reviews to COMMENT while preserving review-ID flow. However, it addresses a different automation issue than the current local PR diff and has blocked regression status. |
| 4 | try-fix-3 |
Blocked | Correctly targets the saved Windows category-filter over-selection issue and includes direct helper probes, but it does not address the XHarness dependency completeness issue and has blocked regression status. |
| 5 | try-fix-1 |
Blocked | Removes formal AI review events by moving summaries to issue comments, but it has broader downstream compatibility risk due to artifact/output variable renames and has blocked regression status. |
Regression-test ranking rule
No saved candidate has a passing regression-test result. The raw PR gate was skipped because no tests were detected. try-fix-1, try-fix-2, and try-fix-3 are all explicitly marked Blocked because full Pester regression execution was unavailable. No candidate failed regression tests, so no failed candidate had to be ranked below a passing candidate.
Expert reviewer impact
The expert reviewer produced one actionable finding against eng/Versions.props:141: updating the central MicrosoftDotNetXHarnessTestRunnersXunitVersion property does not update the actual device-test runner package used by the MAUI device-test projects because those projects still hard-code an old package version. Applying that feedback creates pr-plus-reviewer, which is strictly more complete than the raw PR for the Android/device-test XHarness update.
Winner
pr-plus-reviewer is the winning candidate. It is the PR fix with targeted expert feedback applied in a sandbox copy, so it stays aligned with the submitted dependency-flow change while addressing the only high-confidence review finding. The STEP 5a try-fix candidates are useful for separate automation issues recorded in earlier logs, but they do not improve the current XHarness dependency update and are all blocked on regression execution.
Future Action — review latest findings
No alternative fix was selected for this run. Review the session findings and CI results before merging.
…60601.2 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26230.4 -> To Version 11.0.0-prerelease.26301.2
…60612.1 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26230.4 -> To Version 11.0.0-prerelease.26312.1
|
/review rerun |
This pull request updates the following dependencies
From https://github.com/dotnet/xharness