Fix remaining outerloop test failures

[Ankit Jain (radical)]

Description

The scheduled outerloop run exposed two independent setup and test-contract regressions:

error: release version 21 not supported
System.TimeoutException: Timeout 30000ms exceeded.
- waiting for GetByRole(AriaRole.Button, new() { Name = "Toggle terminal (`)", Exact = true })

The CLI Maven reactor tests are outerloop-only, but Aspire.Cli.Tests did not advertise its Java requirement, so specialized jobs could run with an older JDK. Dashboard shared-layout initialization also subscribes to terminal updates in Playwright tests that do not configure terminals, where the mock previously threw. Separately, two terminal-dock scenarios still targeted the navigation button intentionally removed by #20528.

Mark Aspire.Cli.Tests as requiring Java only when RunOuterloopTests=true, return a completed empty terminal stream from the shared Dashboard mock, and update the terminal-dock scenarios to use the supported AppHost activation path and a surviving settings-button focus target.

Validation:

• CiWorkflowTests: 7 passed.
• Dashboard accessibility matrix: 13 passed.
• CLI outerloop jobs: all 14 tests passed on macOS, Ubuntu, and Windows.
• TerminalDockTests: all 7 outerloop cases passed locally on macOS arm64.

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
  • No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
  • No
• Did you add public API?
  • Yes
  • No
• Does the change make any security assumptions or guarantees?
  • Yes
  • No

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 20457

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 20457"

[Copilot]

Copilot review overview

🟢 Approval recommended

The targeted fixes are consistent with the surrounding infrastructure and no unresolved correctness issues were found.

Review effort: Balanced
Findings: None

What changed in this PR

Fixes scheduled outerloop failures in CLI and Dashboard tests.

Changes:

• Provisions JDK 21 for CLI outerloop tests with focused metadata coverage.
• Provides an empty terminal stream in the Dashboard Playwright fixture.
• Restricts known Fluent UI accessibility exemptions to affected rules and nodes.

File	Description
tests/​Infrastructure.Tests/​WorkflowScripts/​CiWorkflowTests.cs	Verifies CLI outerloop Java metadata.
tests/​Aspire.Dashboard.Tests/​Integration/​Playwright/​Infrastructure/​MockDashboardClient.cs	Returns an empty terminal update stream.
tests/​Aspire.Dashboard.Tests/​Integration/​Playwright/​AccessibilityTests.cs	Adds scoped Fluent UI accessibility allowances.
tests/​Aspire.Cli.Tests/​Aspire.Cli.Tests.csproj	Requires Java for outerloop execution.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Ankit Jain (radical)]

[automated] Extended CI confidence requires action

Reviewer actions

• Required before merge: Fix or revert the empty terminal-subscription behavior, then rerun the final-head Dashboard outerloop job.
  Done when: the Ubuntu Dashboard job executes non-zero tests and all five affected terminal cases pass on the first attempt, while the accessibility cases remain green.

Change classification

• Outerloop workload metadata changed in Aspire.Cli.Tests.csproj so full outerloop runs provision Java for Maven reactor coverage.
• Dashboard outerloop behavior changed through scoped accessibility exemptions and MockDashboardClient.SubscribeTerminalsAsync.
• A focused CI contract test changed; internal Azure DevOps, release/publishing, and deployment E2E paths are not changed by this incremental PR.

🔴 Dashboard terminal outerloop stability — Final-head validation exposed five new failures

• Why it applies: MockDashboardClient.SubscribeTerminalsAsync changed from throwing NotImplementedException to returning an immediately completed empty stream.
• Evidence: In final-head Outerloop Tests run 36074074786, the first Ubuntu Dashboard attempt executed 85 tests and failed five cases after each timed out in TerminalTests.OpenTerminalAsync waiting for a terminal connection:
  • ProducerOutput_RendersInCanvasAndPublicClientSnapshot
  • ModifiedF6_DoesNotMoveFocusFromTerminal
  • FontSizeControls_TakePrimaryAndRespectPackageBounds with Increase font size
  • FontSizeControls_TakePrimaryAndRespectPackageBounds with Decrease font size
  • InitialConnection_UsesProducerDimensions
• Gap or disposition: A later automatic retry passed the Ubuntu Dashboard job, but the same final-head revision was not repeatably green. The five correlated failures are new relative to scheduled baseline run 35946233512 and align with the changed terminal stream, so the retry does not close the regression risk.
• Action: Correct the terminal subscription behavior and rerun the Dashboard outerloop job. Additional pr-testing work is unnecessary until the implementation changes; the existing full outerloop dispatch is the authoritative validation path.

🟡 Final-head outerloop result — Red jobs are inherited outside the incremental diff

• Evidence: The completed final-head outerloop run remained red after automatic retries because of:
  • Ubuntu Templates: 3/3 failed — StarterTemplateWithRedisCacheTests.ResourcesShowUpOnDashboard and both ApiServiceWorks URL variants.
  • Ubuntu CLI E2E: 15/16 passed — RadiusDeployTests.DeployRadiusRedisRabbitMqAndMongoBackingResourcesToKind failed.
  • Windows Dashboard: 82/85 passed with 2 skipped — DashboardInteractionsTests.ScrollButtons_SmoothScrollHidesImmediately_AndReachesNewBottom failed.
  • Final Results failed as the aggregate of those jobs.
• Gap or disposition: Each failure has the same test and failure shape in scheduled baseline run 35946233512, before this incremental diff. None touches the four changed files, so these are inherited/pre-existing rather than PR-caused.
• Action: No incremental-PR action is required for these failures; reviewers should not attribute the aggregate red result to this PR's Java or accessibility changes.

🟢 CLI Java outerloop coverage — Covered on all configured operating systems

• Why it applies: The CLI test project now sets RequiresJava when RunOuterloopTests is true.
• Evidence: The final-head full runsheet contained 25 entries and marked all three CLI jobs requiresJava: true. Their logs show Temurin JDK 21 setup, and each macOS, Ubuntu, and Windows CLI job executed 13 tests with 13 passed, 0 failed, and 0 skipped.
• Gap or disposition: Verified. This directly covers the prior Maven failure, release version 21 not supported, with non-zero final-head test execution.

🟢 Dashboard accessibility exemptions — Covered by non-zero final-head outerloop execution

• Why it applies: AccessibilityTests.cs adds scoped Fluent UI ARIA exemptions.
• Evidence: Final-head Dashboard jobs executed 85 tests on both Ubuntu and Windows. None of the accessibility cases that failed in scheduled baseline run 35946233512 failed on the final head; the latest Ubuntu retry completed successfully.
• Gap or disposition: Verified for the changed accessibility behavior. The remaining Windows failure is the inherited interaction timeout accounted for above.

🟢 Regular PR CI and selector contract — Covered

• Evidence: Final-head CI run 36071887186 selected and passed Aspire.Cli.Tests, Aspire.Dashboard.Tests, and Infrastructure.Tests; Infrastructure.Tests passed 1,356/1,356, including the changed workflow contract test. Final Results and Tests / Final Test Results were green.
• Gap or disposition: Verified for regular tests and selector wiring. Regular PR CI does not replace the specialized outerloop evidence above.

🟢 Quarantine dispatch — Covered

• Evidence: Final-head Quarantine run 36074077921 completed successfully with no red jobs.
• Gap or disposition: Verified; no quarantine-only regression surfaced.

⚪ Internal Azure DevOps and deployment E2E — Not applicable

• Why it does not apply: The exact incremental diff from 9ae5cc2f4e168f41c38659062ecc872bd86947ee to e07e61d44bd2e0e935899585d22cd547e270ad50 changes only three test files and one test project file. It does not change definition-1602 inputs, Arcade/build/release plumbing, packages or artifact layouts, deployment scenarios, or their runtime dependencies.
• Disposition: No internal Azure DevOps or deployment E2E run is required for this incremental PR.

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[Ankit Jain (radical)]

[automated] # PR Testing Report

PR Information

• PR: Fix remaining outerloop test failures #20457
• Title: Fix remaining outerloop test failures
• Head commit: f0af7bac1963c339103df5f01731c9b3873614f7
• Outerloop run: https://github.com/microsoft/aspire/actions/runs/36461436207

Change Validated

• Removed CliOuterloopTestsInstallJava, which asserted an exact XML string in Aspire.Cli.Tests.csproj.
• Added a comment beside RequiresJava explaining that the CLI outerloop Maven fixtures target Java 21.

Local Validation

dotnet test --project tests/Infrastructure.Tests/Infrastructure.Tests.csproj \
  --no-launch-profile -- \
  --filter-class "*.CiWorkflowTests" \
  --filter-not-trait "quarantined=true" \
  --filter-not-trait "outerloop=true"

Result: 5 passed, 0 failed, 0 skipped.

Outerloop Validation

The manually dispatched workflow ran against the expected head commit.

• The generated runsheet set requiresJava: true for the macOS, Ubuntu, and Windows CLI jobs.
• The Set up Java step completed successfully in all three CLI jobs.
• Each CLI job executed 14 tests and passed all 14.
• Both Java-dependent Maven reactor tests passed on all three operating systems:
  • ApplyToRuntimeSpec_ForMavenReactor_StagesAnUninstalledSiblingFromAnEmptyLocalRepository
  • ApplyToRuntimeSpec_ForIndependentMavenReactorModule_AnchorsTheUpstreamInstallAtTheAggregator

Aggregate Workflow Result

The full workflow concluded with failure because other jobs were red:

• Dashboard on Windows and Ubuntu
• CLI end-to-end on Ubuntu
• Templates on Ubuntu

The CLI Java validation targeted by this change passed completely. The aggregate failures require separate attribution because the PR also contains Dashboard changes and the failed jobs are outside the two-file revision made here.

Result

CLI Java outerloop behavior verified; full outerloop workflow remains red from other jobs.

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[Ankit Jain (radical)]

[automated] Final-head review and outerloop validation report.

PR #20457 Review and Outerloop Test Report

PR information

• PR: Fix remaining outerloop test failures #20457
• Title: Fix remaining outerloop test failures
• Head commit: f9c7a630007c04ede57e973e929713608c0d009b
• Outerloop run: https://github.com/microsoft/aspire/actions/runs/36763467633
• Final-head CI: https://github.com/microsoft/aspire/actions/runs/36783635087
• Result: Required PR checks passed; the outerloop run failed after four attempts because of failures also present on main

Final diff reviewed

• tests/Aspire.Cli.Tests/Aspire.Cli.Tests.csproj
  • Sets RequiresJava=true only when RunOuterloopTests=true.
• tests/Aspire.Dashboard.Tests/Integration/Playwright/Infrastructure/MockDashboardClient.cs
  • Returns a completed empty terminal subscription instead of throwing when generic Playwright tests initialize the shared dashboard layout.

Review result

No blocking correctness findings. Both remaining changes are relevant and appropriately scoped.

Conditional CLI Java provisioning

The outerloop runsheet emitted requiresJava: true for the CLI jobs while retaining
/p:RunOuterloopTests=true. The macOS, Ubuntu, and Windows jobs each:

• Installed Java 21.
• Executed 14 tests.
• Passed 14, failed 0, skipped 0.

The previous failure signature was absent:

error: release version 21 not supported

The condition avoids provisioning Java in regular CLI jobs, whose filters exclude the
Java-dependent Maven reactor outerloop tests.

Dashboard mock terminal subscription

Scheduled main runs logged NotImplementedException from
MockDashboardClient.SubscribeTerminalsAsync when the shared layout initialized
TerminalDock.

The generic Playwright mock does not provide terminal descriptors. Returning a
completed empty stream is sufficient for tests unrelated to terminals: it prevents the
layout subscription from throwing without fabricating terminal updates or requiring a
long-lived subscription. Dedicated terminal tests configure their own terminal clients
and connection resolver.

Across all four outerloop workflow attempts, the original NotImplementedException /
SubscribeTerminalsAsync failure signature was absent.

Focused and final-head validation:

• Dashboard accessibility matrix: 13 passed locally.
• Final-head regular PR CI: passed.
• Final-head lint and required aggregate checks: passed.

The first final-head macOS CLI attempt reached the global 10-minute hang-dump timeout
with 5,558 tests passed, 0 failed, and three unrelated tests still running. The failed
jobs passed on rerun; RunOuterloopTests was false, so this PR's Java condition was not
active in that job.

Remaining outerloop failures

CLI end-to-end

All four attempts failed:

Aspire.Cli.EndToEnd.Tests.RadiusDeployTests.DeployRadiusRedisRabbitMqAndMongoBackingResourcesToKind

The same test failed in recent scheduled main runs 36511652601 and 36659156328.
It is unrelated to either file in this PR.

Dashboard Playwright

The fourth attempt executed 85 tests on each operating system:

• Ubuntu: 80 passed, 3 failed, 2 skipped.
• Windows: 80 passed, 3 failed, 2 skipped.

The repeated reported test failures were:

TerminalDockTests.Removal_DoesNotStealFocusAfterMovingElsewhere(hideDock: True)
TerminalDockTests.EmptyDock_ResizingPreservesContentInPriorityOrder

Both fail on recent scheduled main runs 36511652601 and 36659156328. Earlier PR
attempts also produced varying terminal and resource-selection timeouts, matching the
existing flaky pattern rather than a deterministic regression from the completed mock
subscription.

Conclusion

The PR's two fixes are validated and the remaining outerloop failures should not block this PR.

The Dashboard terminal failures and Radius deployment failure reproduce outside this
diff and should be investigated separately. No additional fix belongs in PR #20457
based on this run.

[Copilot]

Copilot review overview

🟢 Approval recommended

The changes correctly address both failures with focused scope and appropriate regression coverage.

Review effort: Balanced
Findings: None

[Copilot]

Copilot review overview

🟢 Approval recommended

Both targeted fixes align with existing CI and dashboard test infrastructure without unresolved issues.

Review effort: Balanced
Findings: None

[Copilot]

Copilot review overview

🔵 Needs a closer look

The actual scope is undocumented, the new action SHA lacks allow-list confirmation, and official pipeline changes lack required internal validation.

Review effort: Balanced
Findings: None

[github-actions]

Tests selector

2 / 97 PR test projects · 0 PR jobs, from 3 changed files.

Selected PR test projects (2 / 97)

Aspire.Cli.Tests, Aspire.Dashboard.Tests

Selected PR jobs (0)

none

---

How these were chosen — grouped by what changed

🧪 tests/Aspire.Cli.Tests/Aspire.Cli.Tests.csproj (changed test)
→ 1 directly: Aspire.Cli.Tests

🧪 tests/Aspire.Dashboard.Tests/Integration/Playwright/Infrastructure/MockDashboardClient.cs (changed test)
→ 1 directly: Aspire.Dashboard.Tests

🧪 tests/Aspire.Dashboard.Tests/Integration/Playwright/TerminalDockTests.cs (changed test)
→ 1 directly: Aspire.Dashboard.Tests

Job reasons

none

---

Selection computed for commit dc6669c.

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[James Newton-King (JamesNK)]

Reviewed commit dc6669c. No new high-confidence issues found in the changed code.

[aspire-repo-bot]

✅ No documentation update needed.

Step 5 branch taken: docs_optional → test_only

Triggered signals: none (signal_count: 0, triggered_signals: []). recommendation was docs_optional.

Allowlist justification: all 3 changed files match tests/**:

• tests/Aspire.Cli.Tests/Aspire.Cli.Tests.csproj
• tests/Aspire.Dashboard.Tests/Integration/Playwright/Infrastructure/MockDashboardClient.cs
• tests/Aspire.Dashboard.Tests/Integration/Playwright/TerminalDockTests.cs

This matches only_test_or_build_changes: true with no source files changed, satisfying the test_only category exactly. The PR fixes outerloop test infra (Java requirement gating for CLI Maven reactor tests, a Dashboard terminal-stream test mock, and terminal-dock test scenarios updated after an unrelated navigation-button removal in #20528). No user-facing product behavior, API, CLI, or dashboard UI changed — only test code and test project configuration. No documentation update is warranted.

[Jose Perez Rodriguez (joperezr)]

LGTM. Thanks Ankit Jain (@radical) !