System.Net.Sockets.Tests.SocketOptionNameTest.BindDuringTcpWait_Succeeds failing in CI

[ahsonkhan]

https://dev.azure.com/dnceng/public/_build/results?buildId=316633 from dotnet/corefx#40446 on netcoreapp-Linux-Debug-x64-Ubuntu.1804.Amd64.Open

System.Net.Sockets.Tests.SocketOptionNameTest.BindDuringTcpWait_Succeeds

System.Net.Sockets.SocketException : Address already in use

Stack trace
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName) in /_/src/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 5202
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress) in /_/src/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 762
   at System.Net.Sockets.Socket.Bind(EndPoint localEP) in /_/src/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 703
   at System.Net.Sockets.Tests.SocketOptionNameTest.BindDuringTcpWait_Succeeds() in /_/src/System.Net.Sockets/tests/FunctionalTests/SocketOptionNameTest.cs:line 445

cc @davidsh, @wfurt

[wfurt]

I found 30 failures in last 120 days. That particular test is not safe to run in parallel with other tests. It will close port and it will try to re-open it again. However with other tests running, some other test can grab it. And when this happen, second bind will fail.

We could move it to outerloop to avoid impact on PRs in other areas or we can disable or remove the test completely. I don't know if there is any way to tell XUnit to not run this particular test in parallel with any other tests. Running with `-parallel none' increases execution time from 50s to 2:30 on my system.

cc: @davidsh @eerhardt @stephentoub

[stephentoub]

I don't know if there is any way to tell XUnit to not run this particular test in parallel with any other tests.

xunit/xunit#318 (comment)
Hopefully we have a version of xunit with that feature implemented.
cc: @ViktorHofer

[wfurt]

https://xunit.net/docs/running-tests-in-parallel.html#parallelism-via-configuration

From what I understand is that test in same collection would not run in parallel but they can still run in parallel with other sets.

[ViktorHofer]

Hopefully we have a version of xunit with that feature implemented.

Sure, we are on the latest. To disable parallelization you have multiple options:

• To disable it for the whole assembly, set <XUnitMaxThreads>1</XUnitMaxThreads> in the csproj
• To disable it for different classes you can use custom test collections: [Collection("Our Test Collection dotnet/corefx#1")]. This goes on the class.
• Generally, tests in the same class don't run in parallel.

[wfurt]

In this case, we can be impacted by almost any other tests - including tests from other areas - like HTTP. From the list above it seems like forcing no parallelism on whole set is only option. Or disabling or removing this specific test.

[stephentoub]

@wfurt, have you tried @bradwilson's suggestion from xunit/xunit#318 (comment)? The whole point of that issue was exactly the case of being able to exclude individual tests from running in parallel with anything else in the assembly... if that fix doesn't actually fix it, Brad should know about it.

[wfurt]

I can give it more try. What documentation describes is somewhat different from that linked issue.

[ViktorHofer]

Happened again: https://dnceng.visualstudio.com/public/_build/results?buildId=335436&view=ms.vss-test-web.build-test-results-tab&runId=9884146&resultId=158598&paneView=debug

Configuration: netcoreapp-Linux-Release-arm64-(Alpine.38.Arm64.Open)Ubuntu.1604.Arm64.Docker.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.8-helix-arm64v8-a45aeeb-20190620184035

System.Net.Sockets.Tests.SocketOptionNameTest.BindDuringTcpWait_Succeeds

System.Net.Sockets.SocketException : Address in use

   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName) in /_/src/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 5013
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress) in /_/src/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 744
   at System.Net.Sockets.Socket.Bind(EndPoint localEP) in /_/src/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 688
   at System.Net.Sockets.Tests.SocketOptionNameTest.BindDuringTcpWait_Succeeds() in /_/src/System.Net.Sockets/tests/FunctionalTests/SocketOptionNameTest.cs:line 445

[karelz]

Next step: Disable the test until we have solution.

[bradwilson]

@ViktorHofer wrote:

Sure, we are on the latest. To disable parallelization you have multiple options:

• To disable it for the whole assembly, set <XUnitMaxThreads>1</XUnitMaxThreads> in the csproj

Assuming that this value is somehow fed into maxParallelThreads of 1, then this disables parallelization of execution, but that doesn't necessarily stop things from being started simultaneously. Example:

• Test 1 starts, opens up to listen on port 1234, and then yields the thread with await.
• Test 2 starts, opens up to listen on port 1234, and fails.

Additionally, if you do anything like .ConfigureAwait(false) in your tests, that will cause any subsequent code to run on the standard Task thread pool and not on the limited execution thread pool that xUnit.net uses to limit concurrency.

The max parallel threads feature in xUnit.net is designed to keep n threads of execution actively running, not limit the system to starting only n tests. If you wrote 1000 tests that just did await Task.Delay(1000);, the entire execution should only take roughly 1 second (minus book-keeping time), because the tests basically spend their entire life awaiting execution, not executing.

There are only two ways to truly turn off parallelization:

• The coarse-grained solution: "parallelizeAssembly": false in xunit.runner.json (or its equivalent command line option, for some runners), which disables all parallelization for the entire assembly
• The fine-grained solution: the aforementioned [CollectionDefinition("...", DisableParallelization = true)], which disables parallelization for a specific test collection