ActiveIssue not skipped on CI for iOS/tvOS devices in Release Configuration

[mdh1418]

System.Collections.Generic.Tests.ComparerTests.NullableComparisons<Int32> is not being skipped on CI as evidenced by this build result even though there is an ActiveIssue attribute

runtime/src/libraries/System.Collections/tests/Generic/Comparers/Comparer.Tests.cs

Line 238 in 41e8677

[ActiveIssue("https://github.com/dotnet/runtime/issues/58933", TestPlatforms.iOS | TestPlatforms.tvOS)]

Oddly enough, running the suite locally on iOSSimulator and on iOS (local iPad), the test is skipped.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[mdh1418]

Looks like the test started failing since 9/17? https://dev.azure.com/dnceng/public/_test/analytics?definitionId=924&contextType=build scrolled down and clicked on System.Collections.Generic.Tests.ComparerTests.NullableComparisons
Not sure why it would be considered a pass before then if it is supposed to be skipped.

[mdh1418]

This is the first instance of the test failing (ActiveIssue not being skipped)
Which seems to point at #57326, and it has been flakey (in skipping this ActiveIssue attribute)

[mdh1418]

Filtered on iOS arm64 and tvOS arm64 test runs. Unfortunately, for iOS and tvOS there will not be a passed test because the test should be skipped if the attribute is handled properly, but judging by the counts, this behaviour is likely flakey.

Tagging subscribers to this area: @Anipik, @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

---

System.Collections.Generic.Tests.ComparerTests.NullableComparisons<Int32> is not being skipped on CI as evidenced by this build result even though there is an ActiveIssue attribute

runtime/src/libraries/System.Collections/tests/Generic/Comparers/Comparer.Tests.cs

Line 238 in 41e8677

[ActiveIssue("https://github.com/dotnet/runtime/issues/58933", TestPlatforms.iOS | TestPlatforms.tvOS)]

Oddly enough, running the suite locally on iOSSimulator and on iOS (local iPad), the test is skipped.

Author:	mdh1418
Assignees:	-
Labels:	area-Infrastructure-libraries, untriaged
Milestone:	-

[mdh1418]

Looks like this is a Release configuration specific problem. on iOS arm64 release, tests are failing, and this ActiveIssue is being skipped, the reason why only this test was caught was because we are only running a small subset of test suites at the moment for devices. When adding an [ActiveIssue("", TestPlatforms.iOS)] to a test in System.Buffers, the test is still ran when in iOS arm64 Release.

[safern]

We don't do much special logic for ActiveIssue, we just check if we are running on that platform:
https://github.com/dotnet/arcade/blob/e43b85e9b4e9ca62b89e407acdf0e56a2cbfd81a/src/Microsoft.DotNet.XUnitExtensions/src/DiscovererHelpers.cs#L28

https://github.com/dotnet/arcade/blob/e43b85e9b4e9ca62b89e407acdf0e56a2cbfd81a/src/Microsoft.DotNet.XUnitExtensions/src/DiscovererHelpers.cs#L29

So maybe something else is happening in RuntimeInformation.IsOSPlatform?

cc: @steveisok

[steveisok]

IsOSPlatform looks good. We're trimming in our test runs, so we're trying to see what things are like w/o it.

[steveisok]

No better... An untrimmed app build in release fails the same way. Passes on debug.

[steveisok]

I found why and we should be able to work around it.

https://github.com/dotnet/arcade/blob/9ab314e39dc623d53c3c7a63a6f63772b01ff6b3/src/Microsoft.DotNet.XUnitExtensions/src/DiscovererHelpers.cs#L17

This returns false because in release mode, Environment.Version.Major returns 0 and RuntimeInformation.FrameworkDescription returns ".NET". It's not clear why the version returns 0.

[steveisok]

We'll need an arcade bump in order to take dotnet/arcade#7965

That should fix the problem.

[safern]

This returns false because in release mode, Environment.Version.Major returns 0 and RuntimeInformation.FrameworkDescription returns ".NET". It's not clear why the version returns 0.

Looks like a bug when running on iOS/tvOS?