Add opt-in fully qualified test name reporting to Helix job monitor

[Evangelink]

Problem

The Helix job monitor reports each test to Azure DevOps using the framework-provided display name as both the visible title and the stable automatedTestName. That is problematic (see dotnet/sdk#55123):

• MSTest reports only the method name, so Tests.ClassA.MyTest and Tests.ClassB.MyTest both appear as MyTest — and are even merged together because aggregation groups by the display name.
• xUnit tests using a custom [Fact(DisplayName = "...")] get an arbitrary, non-unique name that is unstable over time — a poor choice for a test identity.

Change

Add an opt-in mode that reports fully qualified test names. When enabled, the monitor:

• uses the fully qualified name (Namespace.Type.Method) as the stable automatedTestName, keeping a consistent identity/history even when the display name changes;
• groups results by the fully qualified name, fixing same-method-name cross-class collisions;
• formats the visible title as:
  • Namespace.Type.Method when the display name is just the method name (common default),
  • Namespace.Type.Method ("net10.0") for parameterized rows — keeps the argument list without duplicating the method name (handles the case where the FQN does not end with the display name because of the args part),
  • Namespace.Type.Method (My custom name) when a custom display name adds information.

No framework detection is required: the readers already extract type + method separately for trx/xUnit/JUnit, so the FQN is computed deterministically. Legacy behavior is fully preserved when the option is off (default), so existing pipelines are unaffected.

How to enable

jobs:
- template: /eng/common/core-templates/job/helix-job-monitor.yml@self
  parameters:
    useFullyQualifiedTestName: true

Equivalent tool flag --use-fully-qualified-test-name or env var HELIX_USE_FULLY_QUALIFIED_TEST_NAME=true.

Files

• Model/TestResult.cs — derived FullyQualifiedName.
• ResultAggregator.cs — FQN plumbed through AggregatedResult; Aggregate(results, useFullyQualifiedName) groups by FQN when enabled (legacy overload kept).
• TestNameFormatter.cs (new) — the FQN / FQN (args) / FQN (display) formatting.
• AzureDevOpsResultPublisher.cs — applies the option to AutomatedTestName and the title.
• Model/AzureDevOpsReportingParameters.cs, JobMonitorOptions.cs, Services/AzureDevOpsService.cs — option wiring (CLI + env).
• eng/common/core-templates/job/helix-job-monitor.yml — useFullyQualifiedTestName pipeline parameter.
• src/Microsoft.DotNet.Helix/Sdk/Readme.md — documentation.

Tests

New/updated: TestNameFormatterTests, ResultAggregatorTests, LocalTestResultsReaderTests (MSTest trx FQN derivation). 14 passing, build clean (0 warnings/errors).

Addresses the Helix side of dotnet/sdk#55123.

[Copilot]

Pull request overview

This PR adds an opt-in mode for the Helix Job Monitor to report fully qualified test identities (Namespace.Type.Method) to Azure DevOps, addressing collisions/instability caused by framework-provided display names (notably MSTest method-only names and xUnit custom display names).

Changes:

• Introduces UseFullyQualifiedTestName wiring end-to-end (YAML template parameter → tool flag/env var → JobMonitor options → AzDO publisher parameters).
• Adds FullyQualifiedName derivation to parsed test results and switches aggregation/identity to use it when opted in.
• Adds TestNameFormatter to keep titles readable while surfacing the FQN, plus unit tests covering formatter/aggregation/TRX derivation.

Show a summary per file

File	Description
src/Microsoft.DotNet.Helix/Sdk/Readme.md	Documents the new opt-in fully qualified test name reporting mode and how to enable it.
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/TestNameFormatterTests.cs	Adds coverage for the new display-name formatting rules.
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/ResultAggregatorTests.cs	Verifies new FQN derivation and opt-in grouping behavior vs legacy grouping.
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/LocalTestResultsReaderTests.cs	Ensures TRX inputs with method-only names still produce an FQN.
src/Microsoft.DotNet.Helix/JobMonitor/Services/AzureDevOpsService.cs	Plumbs the option into AzDO publisher parameters during upload.
src/Microsoft.DotNet.Helix/JobMonitor/JobMonitorOptions.cs	Adds CLI/env-var configuration for the opt-in behavior.
src/Microsoft.DotNet.Helix/AzureDevOpsTestPublisher/TestNameFormatter.cs	Implements title formatting that preserves FQN while retaining display-name-added info.
src/Microsoft.DotNet.Helix/AzureDevOpsTestPublisher/ResultAggregator.cs	Adds FQN support to aggregated results and opt-in grouping by FQN.
src/Microsoft.DotNet.Helix/AzureDevOpsTestPublisher/Model/TestResult.cs	Derives FullyQualifiedName from type+method with reasonable fallbacks.
src/Microsoft.DotNet.Helix/AzureDevOpsTestPublisher/Model/AzureDevOpsReportingParameters.cs	Adds the opt-in flag to publisher configuration.
src/Microsoft.DotNet.Helix/AzureDevOpsTestPublisher/Microsoft.DotNet.Helix.AzureDevOpsTestPublisher.csproj	Exposes internals to the test project for formatter testing.
src/Microsoft.DotNet.Helix/AzureDevOpsTestPublisher/AzureDevOpsResultPublisher.cs	Uses FQN for automatedTestName and formatted titles when enabled.
eng/common/core-templates/job/helix-job-monitor.yml	Adds useFullyQualifiedTestName parameter and passes the tool flag when set.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 13/13 changed files
• Comments generated: 1

[Copilot]

Copilot's findings

• Files reviewed: 13/13 changed files
• Comments generated: 1

[Copilot]

Copilot's findings

• Files reviewed: 13/13 changed files
• Comments generated: 1

[wfurt]

How does this relates to code in https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter ? This is another chunk that seems to parse test results and use them somewhere.... The change looks good to me in spirit - new behavior guarded by opt-in switch. I don't touch this area often so I do not have full picture here.

[Evangelink]

Good question — they're two implementations of the same "parse test result files → publish to AzDO" step, for two different execution models, and this PR only touches the newer one:

• Sdk/tools/azure-pipelines/reporter (Python) is the legacy in-pipeline reporter. It runs on the Helix machine, inside the work item, as part of the build job that submitted the work (EnableAzurePipelinesReporter). It parses the trx/xunit/junit files right there and pushes results to AzDO.
• AzureDevOpsTestPublisher + JobMonitor (C#) is the decoupled Helix Job Monitor path (EnableHelixJobMonitor, added in Add the Helix Job Monitor #16740). The submitting job returns immediately, and a separate lightweight monitor job later downloads the result files and publishes them. LocalTestResultsReader/ResultAggregator/AzureDevOpsResultPublisher are the C# re-implementation of what the Python reporter does.

They are mutually exclusive by design: when EnableHelixJobMonitor=true, the Sdk disables the Python reporter (no AzDO runs created in the build job) and instead copies the result files to the upload root for the monitor to consume (see the comment in tools/Microsoft.DotNet.Helix.Sdk.props). So a given work item is reported by exactly one of the two.

Why this PR only changes the C# side:

• The bug in Test results in AzDO report unqualified method names after MSTest migration sdk#55123 was observed on the monitor path, which is where the automatedTestName/grouping regression lived (the C# TRX reader dropped the class prefix and grouped by display name).
• The Python TRX reader already qualifies names — formats/trx.py prepends className when the framework only reports the method name (name = classname + '.' + test_name), so it doesn't have the MSTest collision this PR fixes.

Net: this is additive and opt-in on the new monitor path, and it brings that path's TRX handling in line with what the legacy Python reporter already did, plus the extra FQN-as-automatedTestName stability improvement. I intentionally left the Python reporter untouched since it's the legacy path and already behaves correctly here.

[akoeplinger]

why would we not always set this given this is how it used to work with xunit before? is there any downside?

[Evangelink]

For default xUnit it actually makes almost no difference — xUnit's default display name already is the FQN, so the monitor produces the same automatedTestName whether the switch is on or off. The switch only changes two cases: MSTest (whose testName is method-only, the #55123 bug) and any test using a custom DisplayName.

I made it opt-in mainly because automatedTestName is the identity AzDO uses for test history / flakiness / failure grouping, so flipping it is a one-time history discontinuity for anything already reporting through the monitor, and because tests with a deliberate custom DisplayName would see their title change to FQN (display name) and grouping move from display-name to FQN.

If we are fine with taking this small break then I am fine with either dropping the feature flag or switching it on by default (if we want some opt-out)

[wfurt]

I've been working on new HelixClient and trying to remove the Python dependency. I bump to this and added support for AzDo uploader recently. But I;'de be happy to take it out and keep the test runner simple. But I was not sure if this is used by BA/BI anyhow. Any thoughts on this @premun ?
And we would need to solve servicing as well if we want to remove it from Helix. Unlike Arcade that is versioned, Helix queues server all releases at once.

[Evangelink]

I'd appreciate to be able to dogfood on SDK, is there a way to dogfood without having this merged? Or could we merge as-is and do a follow-up to promote as default if everything is good?

[wfurt]

LGTM

[wfurt]

I'm fine taking since it is opt-in. We can sort out longer term strategy separately.

[akoeplinger]

I do think we should change the default. Reporting just the method name as the test name to AzDO is basically useless for a repo that has a lot of tests

[Evangelink]

I do think we should change the default. Reporting just the method name as the test name to AzDO is basically useless for a repo that has a lot of tests

Let me validate it all works well on SDK and I'll do a follow-up PR. Does it sound right?

[akoeplinger]

yep thanks!