Enable Helix job monitor for runtime perf - #132807
Merged
Merged
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 518ace5d-cc70-452b-bb1c-7640ed1674cc
|
Azure Pipelines: Successfully started running 6 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
|
Tagging subscribers to this area: @dotnet/runtime-infrastructure |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the eng/pipelines/performance/perf.yml Azure DevOps pipeline to run the shared Helix Job Monitor job so Helix submissions can complete without tying up build agents, while a dedicated monitor job waits for Helix completion and reports results.
Changes:
- Enables the
enableHelixJobMonitorpipeline variable (overriding the default from/eng/pipelines/common/variables.yml). - Adds the
/eng/common/core-templates/job/helix-job-monitor.ymljob to theBuildstage with a 540-minute timeout andallowNoHelixJobs: true.
lewing
approved these changes
Aug 27, 2026
Member
Author
|
Hold on this until we are sure that the 503s have been resolved. |
Member
|
503s should be resolved now right? |
Member
Author
|
/ba-g unrelated failures |
Member
|
/backport to release/11.0 |
Contributor
|
Started backporting to |
akoeplinger
added a commit
that referenced
this pull request
Sep 9, 2026
akoeplinger
added a commit
that referenced
this pull request
Sep 9, 2026
Backport of #132807 to release/11.0 CI template change, Servicing-approved --------- Co-authored-by: Matt Mitchell <mmitche@microsoft.com> Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com> Copilot-Session: 518ace5d-cc70-452b-bb1c-7640ed1674cc
This was referenced Sep 10, 2026
mmitche
pushed a commit
that referenced
this pull request
Sep 10, 2026
…ine (#133633) ## Summary Fix the regression introduced by **#132807 (Enable Helix job monitor for runtime perf)** by making `DotNet-HelixApi-Access` available at the `Build` stage scope in `eng/pipelines/performance/perf.yml` for internal, non-PR runs. The standalone monitor now inherits the credential needed to discover private Helix jobs. The change is limited to the variable-group import; it leaves the SDK, asynchronous submission, and existing monitor options unchanged. ## Regression The original PR intentionally made `Send job to Helix` return after submission and delegated completion tracking to `Monitor Helix Jobs`. However, the Helix credential group was imported only inside the individual build/submission jobs. The new sibling monitor did not inherit it, leaving its `$(HelixApiAccessToken)` reference unresolved. Helix returns an empty successful discovery response without valid authentication. Combined with `allowNoHelixJobs: true`, this allowed the pipeline to succeed while submitted performance work was still queued. The transition is visible between [20260908.2, before the original PR](https://dev.azure.com/dnceng/internal/_build/results?buildId=3069568&view=logs) and [20260908.3, at its merge commit](https://dev.azure.com/dnceng/internal/_build/results?buildId=3069775&view=logs). Both had the same runtime Arcade SDK pin, so the later SDK update was not the trigger. ## Before / after **Before:** [Monitor log from 20260909.9](https://dev.azure.com/dnceng/internal/_build/results?buildId=3071091&view=logs&j=078edc60-2a90-5618-e72d-426a045b11f0&t=0ca3371c-1de8-59d8-daef-35ea5811e911) showed an all-zero Helix status throughout the run and finished with: ```text Jobs: 0 submitted / 0 resubmitted / 0 processed Work items: 0 submitted / 0 resubmitted / 0 failed ``` **After:** [Internal sanity run 20260910.11 / build 3072330](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072330&view=logs&j=078edc60-2a90-5618-e72d-426a045b11f0&t=0ca3371c-1de8-59d8-daef-35ea5811e911), running this commit with `onlySanityCheck=true`, now shows actual Helix work items and their outcomes in the monitor rather than the all-zero state. For example, the monitor reports `arm64.micro.net11.0.Partition0` from job `40418dd2-e32c-4474-bd8f-e9bf510b7708` and `x86.micro.net11.0.Partition0` from job `4e7b9e80-2a9a-4da8-81a7-731ad8752366`, including their finished state and exit code. This confirms private-job discovery and outcome reporting. **The run is still in progress at the time of opening this PR; full end-to-end completion is not yet claimed.** The sampled ARM64/x86 work-item failures are `NETSDK1045` benchmark-generation failures also present in [main build 3072178](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072178). Two iOS setup jobs encountered a `CertHelper` NuGet restore failure also present in [main build 3072012](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072012). These are separate from the credential-scope fix; the repaired monitor is now surfacing real work-item failures that were previously missed. ## Validation - Parsed the YAML and checked that the only structural difference is the guarded stage-level variable-group import. - `git diff --check`. - Queued internal definition 702 on commit `e6f8fbe40248c99b6ae62ebeef27ff40552d5bea` with `onlySanityCheck=true`; monitoring remains in progress as described above. > [!NOTE] > This PR description and fix were generated with GitHub Copilot. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This was referenced Sep 11, 2026
LoopedBard3
added a commit
that referenced
this pull request
Sep 16, 2026
## Summary Adopt the standalone Helix Job Monitor for `perf-slow`, following #132807, the stage-scoped credential fix in #133633, and the internal-PR credential alignment proposed in #133885. - Always enable asynchronous submission and include the standalone monitor job with an explicit parameter block, including on public and no-selection runs. Existing benchmark selectors are unchanged; no public benchmark workloads are added. - Import `DotNet-HelixApi-Access` at stage scope for all internal-project runs, including PRs, since the sibling monitor does not inherit submitter job variable groups. Always include `helixAccessToken`, using a compile-time `iif` expression to select `$(HelixApiAccessToken)` only for the internal project and an empty string for anonymous access elsewhere. Public runs, including public PRs, receive no private group or token. - Keep a six-hour monitor job cap (`360` minutes; the tool's maximum wait is `355` minutes). Existing submitter and workitem timeouts remain unchanged, so this is not a whole-pipeline wall-clock deadline. - Allow legitimate zero-submission runs and preserve the existing private/scheduled job selection, sanity flag, triggers, and schedules. Only `eng/pipelines/performance/perf-slow.yml` changes. ## Validation Internal pipeline **1012**, [sanity run 3072457](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072457), ran with `onlySanityCheck=true`, `runScheduledJobs=true`, and `runPrivateJobs=true`. - Tested runtime commit: `12fb88f1065a13388d7488fd28e65e18006a08f2`. - Pinned performance resource: `c735123e8b45db1c8d0e30207bfe24a5fca2621c`. - All four build jobs and five Azure submitter jobs succeeded. Sends completed in 29-45 seconds, before their Helix workloads finished. - At 22:23:48 UTC, monitor log **239** recorded nine Azure jobs completed, none running or waiting, while one Helix job/workitem was still running. The monitor was the only remaining Azure job. - After the final workitem finished at 22:24:06 UTC, the monitor drained all results and exited with code 1. Its final summary was **5 jobs submitted, 0 resubmitted, 5 processed; 5 workitems, 5 failed; 5 results uploaded**. The pipeline correctly finished **failed**, not false-green. This demonstrates private job discovery, asynchronous agent release, completion gating, and failure propagation. **It is not a fully passing benchmark run.** ### Failure caveats Four Linux workitems failed with `NETSDK1045`: generated benchmark projects target .NET 12.0 while the SDK reports support through .NET 11.0. The same error was verified in [main's slow-performance run 3072225](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072225), including Linux Helix job `59558d46-2044-4bbe-9b59-cc369574171a`, `arm64.micro.net11.0.Partition0`, console line 848. The Windows workitem failed differently: `WinError 225` blocked `dotnet --info` for the downloaded SDK as a virus or potentially unwanted software. This is **unclassified**, not a confirmed false positive or confirmed pre-existing failure. Evidence: Helix job `071489c5-3f06-4671-b957-0a25abcd5d65`, `arm64.micro.net11.0.Partition0`, console line 436. Build Analysis supplied no known-issue match. ### Final follow-up coverage Subsequent revisions decouple private credentials from monitor eligibility. Commit `1b1a9defdfb3d4a1e1e0b47ba5edaeeb0794ce0e` now includes the flag, job, and all three parameters unconditionally. These changes were **not included in the live run** at `12fb88f1065a13388d7488fd28e65e18006a08f2`. Commit `7b7d4d1bb9319a3f377126a1139e8f60b095db11` also aligns both credential guards with the internal-project-only predicate in #133885 (reference head `0250fbf77bfe09a37256a550a2a3de35414444e9`, open when inspected). It enables credentials for internal PRs while leaving public PRs anonymous. This revision has no new live-run coverage. The final version passed local unique-key YAML parsing and 192 project/reason/private/scheduled/sanity combinations, including 68 public or no-selection cases with a monitor but no benchmark submissions. Checks cover unconditional monitor inclusion, credential import and token selection for internal runs including PRs, an explicit empty token for all other projects, the 360/355-minute limits, preserved workload selection/triggers/schedules, the monitor-template contract, and `git diff --check`. An Azure preview of the earlier unconditional-monitor revision was attempted but denied because `EditBuild` permission is required. No permissions were changed or bypassed. The local expression checks are not an Azure template compiler or a live internal-PR/public-access test, and no new manual CI run or local product build was performed for this pipeline-only revision. > [!NOTE] > This change and PR description were prepared with GitHub Copilot. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com> Copilot-Session: 5c9e5ce3-0d1b-440d-9c69-78918c5f6dc5
jtschuster
pushed a commit
to jtschuster/runtime
that referenced
this pull request
Sep 18, 2026
…#133726) ## Summary Adopt the standalone Helix Job Monitor for `perf-slow`, following dotnet#132807, the stage-scoped credential fix in dotnet#133633, and the internal-PR credential alignment proposed in dotnet#133885. - Always enable asynchronous submission and include the standalone monitor job with an explicit parameter block, including on public and no-selection runs. Existing benchmark selectors are unchanged; no public benchmark workloads are added. - Import `DotNet-HelixApi-Access` at stage scope for all internal-project runs, including PRs, since the sibling monitor does not inherit submitter job variable groups. Always include `helixAccessToken`, using a compile-time `iif` expression to select `$(HelixApiAccessToken)` only for the internal project and an empty string for anonymous access elsewhere. Public runs, including public PRs, receive no private group or token. - Keep a six-hour monitor job cap (`360` minutes; the tool's maximum wait is `355` minutes). Existing submitter and workitem timeouts remain unchanged, so this is not a whole-pipeline wall-clock deadline. - Allow legitimate zero-submission runs and preserve the existing private/scheduled job selection, sanity flag, triggers, and schedules. Only `eng/pipelines/performance/perf-slow.yml` changes. ## Validation Internal pipeline **1012**, [sanity run 3072457](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072457), ran with `onlySanityCheck=true`, `runScheduledJobs=true`, and `runPrivateJobs=true`. - Tested runtime commit: `12fb88f1065a13388d7488fd28e65e18006a08f2`. - Pinned performance resource: `c735123e8b45db1c8d0e30207bfe24a5fca2621c`. - All four build jobs and five Azure submitter jobs succeeded. Sends completed in 29-45 seconds, before their Helix workloads finished. - At 22:23:48 UTC, monitor log **239** recorded nine Azure jobs completed, none running or waiting, while one Helix job/workitem was still running. The monitor was the only remaining Azure job. - After the final workitem finished at 22:24:06 UTC, the monitor drained all results and exited with code 1. Its final summary was **5 jobs submitted, 0 resubmitted, 5 processed; 5 workitems, 5 failed; 5 results uploaded**. The pipeline correctly finished **failed**, not false-green. This demonstrates private job discovery, asynchronous agent release, completion gating, and failure propagation. **It is not a fully passing benchmark run.** ### Failure caveats Four Linux workitems failed with `NETSDK1045`: generated benchmark projects target .NET 12.0 while the SDK reports support through .NET 11.0. The same error was verified in [main's slow-performance run 3072225](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072225), including Linux Helix job `59558d46-2044-4bbe-9b59-cc369574171a`, `arm64.micro.net11.0.Partition0`, console line 848. The Windows workitem failed differently: `WinError 225` blocked `dotnet --info` for the downloaded SDK as a virus or potentially unwanted software. This is **unclassified**, not a confirmed false positive or confirmed pre-existing failure. Evidence: Helix job `071489c5-3f06-4671-b957-0a25abcd5d65`, `arm64.micro.net11.0.Partition0`, console line 436. Build Analysis supplied no known-issue match. ### Final follow-up coverage Subsequent revisions decouple private credentials from monitor eligibility. Commit `1b1a9defdfb3d4a1e1e0b47ba5edaeeb0794ce0e` now includes the flag, job, and all three parameters unconditionally. These changes were **not included in the live run** at `12fb88f1065a13388d7488fd28e65e18006a08f2`. Commit `7b7d4d1bb9319a3f377126a1139e8f60b095db11` also aligns both credential guards with the internal-project-only predicate in dotnet#133885 (reference head `0250fbf77bfe09a37256a550a2a3de35414444e9`, open when inspected). It enables credentials for internal PRs while leaving public PRs anonymous. This revision has no new live-run coverage. The final version passed local unique-key YAML parsing and 192 project/reason/private/scheduled/sanity combinations, including 68 public or no-selection cases with a monitor but no benchmark submissions. Checks cover unconditional monitor inclusion, credential import and token selection for internal runs including PRs, an explicit empty token for all other projects, the 360/355-minute limits, preserved workload selection/triggers/schedules, the monitor-template contract, and `git diff --check`. An Azure preview of the earlier unconditional-monitor revision was attempted but denied because `EditBuild` permission is required. No permissions were changed or bypassed. The local expression checks are not an Azure template compiler or a live internal-PR/public-access test, and no new manual CI run or local product build was performed for this pipeline-only revision. > [!NOTE] > This change and PR description were prepared with GitHub Copilot. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com> Copilot-Session: 5c9e5ce3-0d1b-440d-9c69-78918c5f6dc5
akoeplinger
added a commit
that referenced
this pull request
Sep 21, 2026
## Description Backport the Helix job monitor integration from main/release/11.0, including the groundwork from #129690 and #132150 and the re-enablement in #131969. Includes the subsequent fixes for empty stages (#132019), conditional monitor inclusion (#132882, #132884), parameter forwarding (#133002), performance monitoring (#132807, #133480), and internal credentials (#133633, #133688, #133885). - Use `Microsoft.DotNet.Helix.JobMonitor` version `10.0.0-beta.26461.103`, matching release/10.0's existing Arcade/VMR build. - Preserve the existing SDK, shared Arcade templates, queues, and release/10.0 job layouts. - Follow upstream enablement, except scheduled libraries outerloop runs retain release/10.0's existing warning-only reporting policy. - Leave SuperPMI's post-Helix processing unchanged. The separate perf-slow enablement in #133726 is not included. ## Customer Impact CI infrastructure only; no shipped runtime changes. Moves Helix waiting and test-result reporting into the standalone monitor for the enabled pipelines. ## Regression Not a product regression fix; backports existing CI infrastructure and its follow-up fixes. ## Testing - Validated YAML/JSON/XML configuration and preservation of unrelated settings. - Checked 43 entry pipelines, 149 forwarding sites, and 516 public/internal, PR/scheduled/manual, and normal/staging combinations, plus disabled-mode behavior. - Exercised MSBuild child-property forwarding and the pinned SDK's waiting/reporter properties, including environment-based opt-in. - Restored the monitor and verified its CLI compatibility under .NET 10. No product build or live Azure DevOps pipeline execution was performed locally. ## Risk Changes CI scheduling and result reporting, not product behavior. Uses the monitor package matching the branch's existing toolset and retains the release/10.0-specific scheduled outerloop failure policy. End-to-end Azure DevOps validation is still required. > [!NOTE] > This PR description and backport were prepared with GitHub Copilot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: aae9b201-32a8-45eb-ae1a-db80a0ab4156
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dotnet-runtime-perfpipelinedotnet-runtime-perf-buildunchanged because itsruntime-perf-build-jobs.ymltemplate only builds and publishes artifacts; it does not submit work to HelixMotivation
Recent
dotnet-runtime-perfruns contain manySend job to Helixtasks. Enabling the monitor allows those submissions to return without occupying build agents while one generic job tracks completion and reports results.Validation
git diff --checkperf.ymlandperf-build.yml, respectivelydotnet-performancetemplates to confirm onlyruntime-perf-jobs.ymlreachessend-to-helix-step.ymlNote
This pull request description was generated with GitHub Copilot.