Re-enable VS Code extension E2E tests in CI - #19142
Merged
Adam Ratzman (adamint) merged 8 commits intoAug 12, 2026
Merged
Conversation
github-actions
Bot
added
the
area-engineering-systems
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 19142Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 19142" |
Contributor
There was a problem hiding this comment.
Pull request overview
Re-enables VS Code extension E2E tests and restores aggregate-gate enforcement.
Changes:
- Removes the unconditional skip guard.
- Treats unexpectedly skipped E2E runs as failures.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/tests.yml |
Restores E2E execution and gate checks. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Balanced
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
There was a problem hiding this comment.
Review details
Suppressed comments (2)
.github/workflows/extension-e2e-tests.yml:258
- The temporary-disable behavior has no regression coverage even though
extension/src/test/e2eLaunchProfile.test.ts:216-236already source-checks this exact Azure Functions shard. Without assertingdisabledIssueand the prerequisite guard, a later edit can silently re-enable issue #19151 while those tests still pass. Please update that test to pin the disabled marker and guarded install behavior.
disabledIssue: 'https://github.com/microsoft/aspire/issues/19151'
.github/workflows/tests.yml:830
- This comment does not match the current selector configuration.
setup_for_testspassesenforce: 'false'at tests.yml:69, and.github/actions/select-tests/action.yml:19documents that audit mode sets everyrun_*output to true; non-PR events also force the full set. Therefore this job currently runs on unrelated changes too. Describe the condition in terms of the selector output so the aggregate-gate exception remains accurate in both audit and enforce modes.
# - extension_e2e_tests: this job only runs when the PR/push changes the VS Code extension,
# Aspire CLI, or the extension E2E workflow wiring.
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Balanced
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
There was a problem hiding this comment.
Review details
Suppressed comments (1)
extension/src/test/e2eLaunchProfile.test.ts:510
- This lookup resolves the earlier
waitForWorkspaceRediscoveryLoadingcall at line 92, not the call in the gated test. Removing or moving the gated test's loading assertion afterreleasePsSnapshot()would therefore still pass this regression test and allow the CI race to return. Match the unique call and verify every index exists before checking their order.
assert.ok(appHostTree.indexOf('await waitForWorkspaceRediscoveryLoading') < appHostTree.indexOf('discoveryGate.releasePsSnapshot();'));
- Files reviewed: 5/5 changed files
- Comments generated: 0 new
- Review effort level: Balanced
Reverts the `false &&` guard added in microsoft#18547 and restores the two run_extension_e2e skip-checks in the aggregate gate. The job has been dark since 2026-06-29. microsoft#18547 disabled it because the Windows debug-dashboard shard was consistently failing, but it linked microsoft#18412 as the tracker -- and that issue had already been closed as completed on 2026-06-26 by the *previous* re-enable (microsoft#18464). So there is currently no open issue tracking the disable, and nothing surfaces that the shards do not run. This change is a probe: CI on this PR exercises the shards against current main so we can see empirically whether debug-dashboard still fails, rather than inferring it from a six-week-old report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 487b899d-d79d-4349-a84b-a026861bde7f
The re-enable probe showed the non-Functions VS Code extension E2E shards passing, while the Linux Azure Functions shard timed out waiting for e2e-functions to reach Running. Keep the shard visible in the workflow, but leave the Functions fixture disabled with a tracked issue until the Linux startup timeout is root-caused. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Gate the simulated ps and streaming ls CLI calls so the test observes the loading welcome before allowing the running AppHost snapshot and workspace candidate to arrive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adam Ratzman (adamint)
force-pushed
the
reenable-extension-e2e
branch
from
August 7, 2026 20:10
aada9c2 to
7a481aa
Compare
Contributor
There was a problem hiding this comment.
Review details
Suppressed comments (1)
extension/src/test/e2eLaunchProfile.test.ts:510
- This lookup matches the earlier unrelated loading wait in
appHostTree.e2e.test.ts(currently line 92), not the wait in the gated test. As a result, the assertion still passes ifreleasePsSnapshot()is moved before the intended wait, so the regression guard does not enforce the ordering it claims to protect. Scope the indices to the gated test block.
assert.ok(appHostTree.indexOf('await waitForWorkspaceRediscoveryLoading') < appHostTree.indexOf('discoveryGate.releasePsSnapshot();'));
- Files reviewed: 5/5 changed files
- Comments generated: 0 new
- Review effort level: Balanced
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
6 of 7 tasks
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 487b899d-d79d-4349-a84b-a026861bde7f
A shard row is disabled by adding disabledIssue to its matrix entry, which keeps the job visible while the fixture is broken. That only holds if every shard-only step carries the same condition: guarding the Azure Functions Core Tools install alone still runs the suite without its prerequisites, and a shard that runs no tests but reports green is indistinguishable from one that passed unless the job says why. The test parses the workflow and asserts the run-e2e.js step and every matrix.installAzureFunctions step are guarded by !matrix.disabledIssue, and that the skip notice step is not. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Adam Ratzman (adamint)
marked this pull request as ready for review
August 11, 2026 03:17
Adam Ratzman (adamint)
requested a review
from Ella Hathaway (ellahathaway)
as a code owner
August 11, 2026 03:17
Ella Hathaway (ellahathaway)
approved these changes
Aug 11, 2026
4 of 14 tasks
Adam Ratzman (adamint)
merged commit Aug 12, 2026
ff0dbb4
into
microsoft:main
1073 of 1080 checks passed
4 of 14 tasks
Mitch Denny (mitchdenny)
added a commit
that referenced
this pull request
Aug 12, 2026
PR #19146 (dfa7428) added an assertion that every disabled row in the extension E2E workflow matrix is explicitly tracked in expectedDisabledRows. PR #19142 (ff0dbb4) added a disabled row for the Linux azure-functions shard 80 seconds later. Each was green on its own branch, but together they form a semantic merge conflict: the workflow has one disabled row and the allowlist is empty, so "Run VS Code extension unit tests (Windows)" fails on every run of main. Add the tracking entry so the allowlist matches the workflow. The allowlist stays hand-maintained by design - deriving it from the workflow would make the assertion vacuous, since its whole purpose is to force disabling a shard to be an explicit, reviewable code change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 task done
This was referenced Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-enables the VS Code extension E2E workflow by removing the
false &&guard and restoring the aggregate-gate skip checks.Probe result
Run 31202231991 exercised the E2E matrix after the guard was removed.
Verify ExTester internal feed availabilitypassed.Passed on both Linux and Windows in that run:
apphost-tree,command-palette,debug-dashboard,debug-startup-timeout,discovery-configuration,edge-cases,package-surface,settings-files,tree-actions, andzero-to-running.debug-dashboardpassing on both platforms removes the main known reason the job had been disabled.A later run exposed a race in
apphost-tree: the test waited for the transientSearching for AppHosts...welcome after discovery could already finish. This PR now gates the simulatedpsand streaminglsCLI calls so the loading state is observed deterministically before releasing the running AppHost snapshot and then the workspace candidate.The only deterministic E2E shard failure from the original probe was
azure-functionson Linux: it timed out after 300000ms waiting fore2e-functionsto reachRunningwithLast error: <none>. The job log shows Azure Functions Core Tools 4.12.1 was installed andfunc --versionsucceeded, so this was not a missing Core Tools install. The workflow currently has no Windowsazure-functionsshard.Follow-up
The
azure-functionsshard remains visible in the reusable workflow, but the Functions fixture is disabled pending #19151. The other VS Code extension E2E shards are re-enabled.Validation after the AppHost race fix: run 31212569613 passed both
apphost-treejobs (LinuxandWindows) and reported no VS Code extension E2E failures.Review follow-up
Re-checked the guard and fixed it to slice the intended
running AppHosts appear before slow discovery resultstest block before comparing the wait/release ordering. The earlier unrelatedwaitForWorkspaceRediscoveryLoadingoccurrence can no longer make the guard pass.Validation from
extension/:Mutation check: moving the intended wait after
discoveryGate.releasePsSnapshot()made the fixed guard fail on the wait-before-release assertion.Pushed commit:
b74dc620c7df64fb67177c6f5f57590cc4da73f5.Review convergence follow-up
Latest Copilot pass found that
disabledIssueskipped Azure Functions prerequisites but not the actual E2E execution step. TheRun extension E2E testsstep now has the same!matrix.disabledIssueguard, so disabled shards stay visible in the matrix without running the spec.Validation:
Mutation check: removing the
Run extension E2E testscondition madepins the real Azure Functions toolchain for the offline E2E shardfail onassert.ok(runStep.includes('if: ${{ !matrix.disabledIssue }}')). Restored fix reran green.Pushed commit:
0d41da0bf69bd957cc94c659dd74c9dc13395ccb.