[Feature]: xunit's `DisableDiscoveryEnumeration`

[kzrnm]

Problem Statement

It would be useful to have an option similar to xUnit’s DisableDiscoveryEnumeration, where test cases are not discovered during discovery but instead executed at runtime.

This could help reduce IDE overhead when there are thousands of test cases.

https://api.xunit.net/v3/2.0.0/v3.2.0.0-Xunit.TheoryAttribute.DisableDiscoveryEnumeration.html

Proposed Solution

public static IEnumerable<int> TestData() => Enumerable.Range(0, 10000);
[Test]
[MethodDataSource(nameof(TestData), DisableDiscoveryEnumeration = true)]
public async Task MyTest(int input)
{
    await Assert.That(input).IsBetween(0, int.MaxValue);
}

Alternatives Considered

No response

Feature Category

Data-Driven Testing (Arguments, DataSources)

How important is this feature to you?

Nice to have - would improve my experience

Additional Context

No response

Contribution

• I'm willing to submit a pull request for this feature

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[thomhurst]

@kzrnm From what I've seen with IDEs, if we do this, I'm not sure the IDE's test explorer's will update correctly at runtime to reflect the new tests that weren't there at discovery time.

I think they might do some special handling for xUnit, that I don't think is there for Microsoft Testing Platform.

I could still do this, but it might still be a less than ideal UX unless the IDEs fix up their side too

[thomhurst]

@kzrnm This should be in now. From my testing, the IDEs never show more than the original singular "placeholder" test, even after running.

If you see the same, would you mind raising with Jetbrains & Microsoft so we can hopefully get a fix in on Rider and Visual Studio's side

[kzrnm]

the IDEs never show more than the original singular "placeholder" test, even after running.

Individual test results are not displayed in Visual Studio's Test Explorer, as shown below. xUnit displays them individually, so it may be possible to resolve this.

Test

namespace Mtp.Local.TUnit;

public class MultiTests
{
    public static IEnumerable<int> Numbers => Enumerable.Range(0, 4);

    [Test]
    [MethodDataSource(nameof(Numbers), DeferEnumeration = true)]
    public async Task Defer(int number)
    {
        await Assert.That(number % 2).IsEqualTo(0);
    }

    [Test]
    [MethodDataSource(nameof(Numbers))]
    public async Task Normal(int number)
    {
        await Assert.That(number % 2).IsEqualTo(0);
    }
}

[thomhurst]

I'm pushing the tests through the Microsoft.Testing.Platform layer - It's up to the IDE after that. I don't think I can do anything else.

I'm not sure if IDEs have specific logic for the xUnit feature or not? I know in Rider they have a specific xUnit panel.