Switch HelixTestRunner from dotnet test to dotnet vstest

[mmitche]

Switch HelixTestRunner from dotnet test to dotnet vstest

Why

Both commands delegate to vstest.console internally, but dotnet vstest:

• Accepts multiple DLL paths as positional arguments (needed for future batching work in Batch small Helix work items to reduce per-item overhead (-53% compute) #66808)
• Has a more predictable argument format (no ambiguity between project/solution/DLL parsing)
• Is already used for test discovery in CheckTestDiscoveryAsync (dotnet vstest {target} -lt)

Argument mapping

dotnet test	dotnet vstest
--filter "expr"	--testcasefilter:"expr"
--blame-crash --blame-hang-timeout 15m	--blame:CollectDump
--logger name	--logger:name
--diag:path	--diag:"path"

Other changes

• Deduplicates the quarantined/non-quarantined if/else branches into a shared filter variable
• No functional change to test execution — same vstest.console engine, same test results format

[dotnet-policy-service]

Hey @dotnet/aspnet-build, looks like this PR is something you want to take a look at.

[mmitche]

@wtgodbe This is in prep for the batching changes. I wanted to break them down into smaller chunks, and switching to vstest allows for passing of multiple test dlls at once.

[mmitche]

Not yet ready.

[Copilot]

Pull request overview

This PR updates the HelixTestRunner’s test execution path to invoke dotnet vstest instead of dotnet test, aligning execution with the already-used discovery approach and enabling future batching scenarios.

Changes:

• Switches test execution to dotnet vstest with mapped arguments (--testcasefilter, --diag, --logger, blame options).
• Deduplicates quarantined vs non-quarantined execution logic by using a shared filter variable.

[mmitche]

Validated locally that these work.

[Youssef1313]

I'm not sure this is good long-term.

@nohwnd What's the current status of dotnet vstest?

https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-vstest

The dotnet vstest command is superseded by dotnet test, which can now be used to run assemblies. See dotnet test.

Should I read this as "kinda deprecated"?

[nohwnd]

I would like to remove dotnet vstest eventually, so I would not recommend moving to it.

All that dotnet vstest can do, dotnet test should be able to do as well. As soon as you give it a single (or more) dll parameters it will switch to "dotnet vstest" mode. Apart from the issues listed in the description is there anything else?

• Accepts multiple DLL paths as positional arguments - dotnet test does that as well
• Has a more predictable argument format - is that significant advantage when you are running it programatically?
• Is already used for test discovery - dotnet test has the same parameter that works the same way.

Since you talk about discovering tests? What is the task here that is being achieved? If you are orchestrating test runs then maybe running via translation layer (like VS does it) might provide more power to what you are doing.

And give us more push to implement things like sharding of test runs based on historical results, to optimized utilization of servers etc.

[mmitche]

I would like to remove dotnet vstest eventually, so I would not recommend moving to it.

All that dotnet vstest can do, dotnet test should be able to do as well. As soon as you give it a single (or more) dll parameters it will switch to "dotnet vstest" mode. Apart from the issues listed in the description is there anything else?

• Accepts multiple DLL paths as positional arguments - dotnet test does that as well
• Has a more predictable argument format - is that significant advantage when you are running it programatically?
• Is already used for test discovery - dotnet test has the same parameter that works the same way.

Since you talk about discovering tests? What is the task here that is being achieved? If you are orchestrating test runs then maybe running via translation layer (like VS does it) might provide more power to what you are doing.

And give us more push to implement things like sharding of test runs based on historical results, to optimized utilization of servers etc.

Ahh interesting! The models do NOT think that dotnet test can accept multiple dlls. The help and documentation also does not make this obvious.

I'm not sure what the purpose of the discovery is.

[mmitche]

@wtgodbe Do you happen to know what the discovery is for?

[mmitche]

Note that roslyn uses dotnet vstest so we might want to make a change there.

[nohwnd]

👍 we did not do any analysis on impact of removing vstest recently, so no hurry. I expect many places still using it. There should be some telemetry when revisiting.