Shard Windows x86 Process tests to fit the Helix work-item budget

[steveisok]

Summary

Mitigate #135001's Process-suite work-item time-budget exhaustion by partitioning Windows x86 CoreCLR console-runner Helix archives into six independent work items. Each process retains the existing CollectionPerAssembly behavior. No in-process parallelism change, test/fixture alteration, new skip, or timeout increase.

The ordinary local runner remains full-suite. Mono, other architectures/platforms, NativeAOT/ReadyToRun and other single-file runners retain their existing unpartitioned paths. The investigative watchdog/module initializer, forced/snapshot-only dump logic, verbose-only setting and diagnostic README are removed.

Validation in progress: the five-minute elapsed goal has NOT yet been verified in CI. Temporary PR-only CI routing remains below, so this revision is not yet production-ready for merge.

Evidence and balancing

Treat #135001 independently; reusing #131907 is only a convenient branch/PR vehicle. #131389's old StateRepository diagnosis is historical comparison, not the presumed cause of these timeouts.

The snapshot-only exact-leg run 1621730 reached the original 900-second timeout with 632 completed rows and approximately 883.13 seconds of completed test time. Its final unmatched row is not thereby proven hanging. A matched same-architecture/configuration/queue Windows 14393 passing run, build 1618186, completed 691 rows in 640.397 test seconds and 665.216 whole-work-item seconds. Matched rows were broadly slower in the failing run; this supports the budget mitigation, not a machine/runtime root-cause claim or a claim that all historical matches are identical.

Whole classes are balanced using all 691 passing rows, observed slow method floors, conservative 2× passing timing, and 45 seconds of work-item setup/runtime overhead per shard. Duplicate display names are preserved; unobserved slow rows and non-exactly escaped names are not discarded. Five shards' conservative average exceeds 300 seconds; six leave measured-model headroom without dozens of tiny work items.

Work item suffix	Historical rows	Conservative full elapsed forecast
.1	64	257.5s
.2	75	256.7s
.3	222	252.9s
.4	125	256.4s
.5	122	252.6s
.6	83	272.1s

These are forecasts, not achieved CI timings. The acceptance criterion is every Process shard's observed whole-work-item elapsed ≤300s, including setup, with complete exactly-once coverage. Any over-budget shard must be rebalanced.

Implementation and coverage

ProcessTestShards.targets customizes only this project's archive generation using the repository's existing GenerateRunScript and ZipDirectory tasks. Default Helix packaging already maps each ZIP to a separate named work item; no new CI runner or shared infrastructure refactor is introduced.

Each archive contains the unchanged full test assembly, runtimeconfig/deps, RemoteExecutor and supporting files, with class selection in its generated runner. Work items are System.Diagnostics.Process.Tests.1 through .6, each with independent extracted payload, result/upload directory, temp scope and child-process lifecycle. The original unfiltered ZIP is removed after all six archives exist, and unsupported-mode generation removes stale shard ZIPs. The full local RunTests.cmd is not replaced.

The first five shards use include-class lists (OR semantics). The sixth excludes exactly those sixteen classes: it covers the remaining classes and future additions automatically. Theory rows and inherited cases stay with their executing class. All work items retain the original 900-second timeout; there is no artificial five-minute kill.

Scoped validation

• Built only the existing installer.tasks script-generation task: zero warnings/errors. No broad runtime baseline build.
• Invoked actual test-project script/archive generation against the latest exact-leg execution payload. Exactly six named ZIPs were generated, the original ZIP was absent, every archive retained required execution assets, and no recursively nested staging payload was included. Re-ran generation after the final incremental-input correction; changing the full runner remains an archive input.
• Actual sendtohelixhelp.proj generation produced all six expected HelixWorkItems, the standard runtime-path command, and Timeout=00:15:00.
• Discovery-only validation used the latest compiled test payload on its matching x86 runtime and the existing xUnit discovery/filter APIs: 724 discovered / 689 eligible local cases, all assigned exactly once. This local nonprivileged Windows 11 discovery differs from historical Windows 14393's 691 rows because conditional admin tests expand differently: TestWindowStyle yields one skipped theory instead of four rows, while the protected-process theory expands to two instead of one skip. Traits/platform conditions were not changed to force a count.
• Independently verified all 691 historical rows with full duplicate-name multiplicity against the exact declared filters. Current eligible discovery, rather than a hard-coded historical count, is the partition invariant.
• Executed the existing x86 runner on a small synthetic fixture to verify include-OR, exact executing-class identity for inherited facts, theory coverage, new-class complement, empty shard/complement behavior and no duplicate rows.
• Checked normal/local/other-architecture/Mono/single-file fallback gates; actual Mono archive generation stayed full-suite. Full NativeAOT build/evaluation needs the unavailable native baseline; it is excluded by the single-file gate, not by assuming console flags work on a static runner.
• The changed repository Process tests were not built or executed locally: the earlier full-project attempt was blocked by the missing shared-framework baseline. The reused execution payload supports generation/discovery validation, not a claim that newly compiled repository tests passed. CI must supply the new compile, complete coverage and all six elapsed measurements.

Temporary CI routing and remaining gate

The existing build and Helix-submission opt-ins for this PR's merge ref, PullRequest reason, Windows/x86 job variables remain unchanged, ensuring build_windows_x86_Debug_Libraries_CheckedCoreCLR runs on Windows.10.Amd64.Open (windows.10.amd64.open.rt) despite a libraries-only diff. The exact diagnostic runtime.yml path exclusion remains limited to this PR and does not hide genuine CoreCLR changes or alter other PRs' classification.

This routing is temporary validation wiring and must be removed or replaced with justified permanent routing before merge. Do not call the mitigation complete until all intended shard work items have full elapsed ≤300s and their per-run test-row union proves no omissions or duplicates. A successful pipeline alone, predicted balance, or historical fixed count is insufficient.

Note

This pull request description and changes were generated with GitHub Copilot.

[steveisok]

/azp run runtime-libraries-coreclr outerloop-windows

[steveisok]

/azp run runtime-nativeaot-outerloop

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Adds additional diagnostics and run-configuration changes to help investigate hangs in System.Diagnostics.Process tests, primarily on Windows. The PR introduces both logging (per-test + targeted messages) and a watchdog/WER dump configuration mechanism.

Changes:

• Windows-only test execution configuration updates (disable parallelization, enable progress output, and currently filters to a single test class via XUnitOptions).
• New diagnostics helper with a module initializer, WER LocalDumps configuration attempt, and a FailFast watchdog.
• Adds assembly-level xUnit BeforeAfterTestAttribute hook and adds extra diagnostic logging in a couple of ProcessStartInfoTests.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj	Adds Windows-only test runner settings (parallelization/progress and class filtering) and includes the new diagnostics source file.
src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs	New diagnostics implementation: per-test logging hook, module initializer logging, WER LocalDumps setup, and watchdog FailFast timeout.
src/libraries/System.Diagnostics.Process/tests/ProcessStartInfoTests.cs	Adds targeted diagnostic logs around specific test execution points.
src/libraries/System.Diagnostics.Process/tests/AssemblyInfo.cs	Registers the diagnostics attribute at assembly level.

Suppressed comments (1)

src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:83

• Configuring WER LocalDumps under HKLM requires admin and will typically fail in CI/Helix (and if it succeeds, it leaves a machine-wide setting behind). Using HKCU avoids the elevation requirement and limits scope to the current user, making it more likely the dump configuration actually takes effect.

            {
                using RegistryKey? key = Registry.LocalMachine.CreateSubKey(keyPath);
                if (key is null)
                {
                    Log($"Unable to create WER LocalDumps key HKLM\\{keyPath}.");

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
See info in area-owners.md if you want to be subscribed.

[steveisok]

/azp run runtime-libraries-coreclr outerloop-windows

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[steveisok]

/azp run runtime-nativeaot-outerloop

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj:17

• The hard-coded -class System.Diagnostics.Tests.ProcessStartInfoTests filter in XUnitOptions means Windows runs of this test project will only execute that single test class, skipping the rest of the Process test suite on Windows (significant test coverage reduction). This kind of focusing should be passed as a CI/test invocation parameter (e.g., /p:XUnitClassName=...) rather than baked into the project file.

  <PropertyGroup Condition="'$(TargetPlatformIdentifier)' == 'windows'">
    <TestDisableParallelization>true</TestDisableParallelization>
    <XUnitOptions>$(XUnitOptions) -class System.Diagnostics.Tests.ProcessStartInfoTests -parallel none</XUnitOptions>
    <XUnitShowProgress>true</XUnitShowProgress>

src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:49

• The watchdog thread is started unconditionally from the module initializer and will FailFast after 3 minutes even for non-hung runs (e.g., slow machines or local debugging). Consider gating this to Helix runs (or an explicit opt-in env var) so it doesn’t introduce new crash behavior in normal Windows test execution.

            var watchdog = new Thread(Watchdog)
            {
                IsBackground = true,
                Name = "Process tests hang watchdog"
            };
            watchdog.Start();

src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:87

• Configuring WER LocalDumps via Registry.LocalMachine writes machine-wide state and will typically require elevated permissions; if it fails (UnauthorizedAccess), the dump capture won’t be configured at all. Prefer using a per-user key so the diagnostics can work under normal test permissions and avoid persisting HKLM changes.

                using RegistryKey? key = Registry.LocalMachine.CreateSubKey(keyPath);
                if (key is null)
                {
                    Log($"Unable to create WER LocalDumps key HKLM\\{keyPath}.");
                    return;

[steveisok]

/azp run runtime-libraries-coreclr outerloop-windows

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[steveisok]

/azp run runtime-nativeaot-outerloop

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 0 new
• Review effort level: Lite

[noahfalk]

Assuming this isn't expected to stay disabled forever how about we add a comment that links the skip back to issue that tracks investigating and eventually re-enabling the test?

[steveisok]

Yeah, I need to create the infra issue and link it here.

[adamsitnik]

@steveisok Big thanks for investigating the hang! We have refactored the UseShellExecute code in #126314, I wonder if we have introduced a regression (cc @jkotas)

[steveisok]

@adamsitnik I have this issue draft from the investigation - https://gist.github.com/steveisok/2f9b48f8833a7708b061478ffb4bedb8

It seems like it's a windows bug and something we should give to dnceng

[jkotas]

Does not repro anymore #131389 (comment)

[azure-pipelines]

Azure Pipelines:
Successfully started running 6 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[steveisok]

Superseded by #135160, which carries the validated Process-suite sharding change in a clean, focused history without temporary diagnostic CI routing. Closing this investigation vehicle; its evidence remains available here.

Note

This comment was generated with GitHub Copilot.