Skip to content

Gate the CLI starter validation jobs on the test selector - #19807

Merged
Ankit Jain (radical) merged 13 commits into
mainfrom
copilot/enable-cli-starter-validation-selector
Sep 3, 2026
Merged

Ankit Jain (radical) merged 13 commits into
mainfrom
copilot/enable-cli-starter-validation-selector

Conversation

Copilot AI commented Aug 29, 2026 •

Copy link
Copy Markdown
Contributor

[automated] ## Description

The six cross-platform CLI starter validation jobs previously ran on every pull request. They now participate in selective CI and run only when a changed consumer can affect the starter scenarios.

The selector exposes job:cli-starter-validation, threads its boolean through setup_for_tests, gates all six platform jobs, and treats a skipped job as an error only when the selector requested it.

The trigger follows stable direct artifact boundaries instead of selected test projects:

  • Affected-project rules cover the CLI and managed executables, AppHost SDK, TypeScript code generation and JavaScript hosting, AppHost and PostgreSQL hosting packages, and CreateLayout bundle assembly.
  • Exact path rules cover C# starter content and package construction, executable CLI archive definitions, eng/Bundle.proj, and the validation workflow, scenario script, and PR installer.
  • CreateLayout and eng/Bundle.proj route through CLI_BUNDLE, which now includes starter validation.

Selected test projects are intentionally not used as proxies. Their dependency graphs also include unrelated templates, Azure/browser integrations, Npgsql client code, and test-only utilities or fakes, which would fan unrelated changes into all six platform jobs.

Layer 1 still supplies transitive dependency closure behind each direct production boundary, avoiding a granular list of implementation projects. Core Hosting changes therefore reach starter validation, while unrelated Redis, Npgsql client, Azure App Containers, xUnit template, npm-launcher, native-symbol packaging, and test-infrastructure changes remain excluded.

The real-map tests pin the direct runtime inputs, loose package and workflow inputs, executable CLI archive boundaries, bundle construction, and unrelated-change exclusions. A workflow invariant also requires every platform's skipped-result check to remain coupled to run_cli_starter_validation.

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

Co-authored-by: radical <1472+radical@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 19807

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 19807"

@github-actions github-actions Bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Aug 29, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Routes six cross-platform CLI starter validation jobs through selective CI.

Changes:

  • Adds the job:cli-starter-validation selector target and routing rules.
  • Gates workflow jobs and result checks on the selector output.
  • Updates selector tests and documentation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
.github/workflows/tests.yml Adds selector-based job gates.
eng/github-ci/test-trigger-map.yml Defines starter-validation routing.
tests/Infrastructure.Tests/TestTriggerMap/TestTriggerMapTests.cs Tests target and gate wiring.
docs/ci/test-trigger-map.md Documents the new target.
Suppressed comments (1)

eng/github-ci/test-trigger-map.yml:419

  • The new curated project routing is not exercised by a real-map behavior test. The added tests cover the script path and YAML gate binding, while EveryAffectedProjectRuleGlobMatchesASolutionProject only proves that each name matches some project; it would still pass if a required project were omitted or a leaf such as Aspire.Hosting.Redis were added, defeating this PR's selection contract. Add positive Layer 1 cases for representative listed projects and the documented negative Redis case.
  - projects:
      - Aspire.Cli
      - Aspire.TypeSystem
      - Aspire.Managed
      - Aspire.AppHost.Sdk

Comment thread eng/github-ci/test-trigger-map.yml Outdated
@github-actions

This comment has been minimized.

Comment thread eng/github-ci/test-trigger-map.yml Outdated
Co-authored-by: radical <1472+radical@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

eng/github-ci/test-trigger-map.yml:25

  • job:cli-starter-validation is not added to the CLI_BUNDLE group at lines 33-39. Consequently, an eng/Bundle.proj-only change selects the existing bundle consumers but leaves run_cli_starter_validation false, even though these jobs install that bundle. Add this target to CLI_BUNDLE and cover eng/Bundle.proj with a real-map assertion.

This issue also appears in the following locations of the same file:

  • line 246
  • line 415
#   job:cli-starter-validation tests.yml cli_starter_validation_* (6 platforms)

eng/github-ci/test-trigger-map.yml:250

  • This routing only covers the workflow and validation script, but not several loose inputs the job executes: src/Aspire.ProjectTemplates/**, src/Aspire.Cli/Templating/Templates/ts-starter/**, eng/dashboardpack/**, and eng/dcppack/**. Their existing rules select Templates/Hosting SDK/TerminalHost or TypeScript SDK tests, not either CLI test keyed by the new derived rule, so those changes can leave all six starter jobs skipped. Route these paths to job:cli-starter-validation and add exact real-map cases.
  - paths:
      - .github/workflows/cli-starter-validation.yml
      - eng/scripts/cli-starter-validation.ps1
    targets: [job:cli-starter-validation]
    reason: the reusable workflow and the script it runs ARE the starter validation

eng/github-ci/test-trigger-map.yml:417

  • Deriving the job only from selected CLI tests does not implement the required production-project triggers. For example, Aspire.Dashboard and Aspire.Hosting.PostgreSQL do not flow to either CLI test through ProjectReferences, so changes to the dashboard launched by every run or the PostgreSQL package added by the script can skip this validation. Add the intended exact affected_project_rules entry (including the other starter dependencies) and focused real-map coverage.
  - tests: [test:Aspire.Cli.Tests, test:Aspire.Cli.EndToEnd.Tests]
    targets: [job:cli-starter-validation]
    reason: starter validation follows the same trigger set as the CLI and CLI end-to-end tests

@github-actions

This comment has been minimized.

CLI starter validation was derived only from CLI unit and end-to-end
tests. Changes to the C# starter, AppHost/PostgreSQL path, packaged
DCP/dashboard SDK, and CreateLayout could therefore skip all six
platform jobs.

Derive the job from stable consumer tests instead of a production-project
allowlist. Include it in CLI_BUNDLE and route CreateLayout through that
group. Add real-map tests for each consumer class while keeping unrelated
leaf integrations excluded.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5289c147-1640-420e-9f36-10024c2af62e
Copilot AI review requested due to automatic review settings August 30, 2026 03:42
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comment thread eng/github-ci/test-trigger-map.yml Outdated
Selected test projects are not reliable proxies for the starter scenarios.
They omit some direct runtime inputs while also aggregating unrelated
templates, integrations, client libraries, and test-only dependencies.

Route starter validation through narrow affected-project rules for the direct
production artifacts and exact path rules for package, template, and workflow
inputs. Layer 1 still supplies transitive closure behind those stable
boundaries.

Add positive coverage for every direct scenario input and negative coverage
for unrelated integrations, templates, clients, and test infrastructure.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5289c147-1640-420e-9f36-10024c2af62e
Copilot AI review requested due to automatic review settings August 30, 2026 04:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot could not run the full agentic suite for this review because it was automatically requested on a bot-authored pull request. Request a review from Copilot under Reviewers to retry with the full agentic suite. Improved support for bot-authored pull requests is coming soon.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

Comment thread .github/workflows/tests.yml Outdated
Comment thread tests/Infrastructure.Tests/TestTriggerMap/TestTriggerMapTests.cs Outdated
Comment thread tests/Infrastructure.Tests/TestTriggerMap/TestTriggerMapTests.cs Outdated
@github-actions

This comment has been minimized.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

The starter archive-path invariant scraped YAML and embedded JSON with
format-sensitive regexes and deferred assertions. This could fail after
harmless formatting changes or silently inspect only the first matching
archive dependency or RID.

Parse the workflow with YamlDotNet and the targets input with
System.Text.Json, require exactly one archive dependency and target per
starter job, and collect paths eagerly for clearer diagnostics.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5289c147-1640-420e-9f36-10024c2af62e
Copilot AI review requested due to automatic review settings September 2, 2026 20:37
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The cross-platform CI routing changes require final human validation.

Review details
  • Files reviewed: 4/4 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

The final-results gate repeated the same selector output on every starter
platform. Group the six skipped-result checks behind one selection guard
while preserving the explicit platform list.

Update the regression test to pin the complete grouped condition and
require the selection guard to appear exactly once.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5289c147-1640-420e-9f36-10024c2af62e
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Tests selector

Selects the full PR test matrix + all PR-gated jobs (ALL) — a rule matching '.github/workflows/tests.yml' selects ALL

Advisory workflow impact (4)

  • Aspire.Deployment.EndToEnd.Tests (deployment workflow-only)
  • Aspire.EndToEnd.Tests (outerloop-only)
  • Aspire.Oracle.EntityFrameworkCore.Tests (outerloop-only)
  • deployment-e2e (schedule/dispatch-only)

Selection computed for commit 0c7256b.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

All reviewed changes are covered with no unresolved issues.

Review details
  • Files reviewed: 4/4 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@radical
Ankit Jain (radical) merged commit 9fe2f3d into main Sep 3, 2026
1172 of 1178 checks passed
@radical
Ankit Jain (radical) deleted the copilot/enable-cli-starter-validation-selector branch September 3, 2026 03:19
@github-actions github-actions Bot added this to the 13.6 milestone Sep 3, 2026
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

⚠️ CI Failure Analysis: Possible Flaky Test(s)

The CI build failed due to test failure(s) that appear unrelated to the PR changes. These may be flaky tests.

Suspected flaky test(s):

Suggested actions:

  • Re-run the failed CI jobs to confirm if the failure is intermittent
  • If the test continues to fail, consider quarantining it using /quarantine-test <test name> <issue URL>
  • Search existing issues to see if this test is already known to be flaky

You can re-run the failed jobs from the workflow run page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants