Trimming tests fail with race condition

[safern]

/Users/runner/work/1/s/.dotnet/sdk/5.0.100-preview.8.20362.3/Microsoft.Common.CurrentVersion.targets(4187,5): error MSB3026: Could not copy "/Users/runner/work/1/s/artifacts/obj/Microsoft.Extensions.DependencyInjection/net5.0-Release/Microsoft.Extensions.DependencyInjection.dll" to "/Users/runner/work/1/s/artifacts/bin/Microsoft.Extensions.DependencyInjection/net5.0-Release/Microsoft.Extensions.DependencyInjection.dll". Beginning retry 1 in 1000ms. The process cannot access the file '/Users/runner/work/1/s/artifacts/obj/Microsoft.Extensions.DependencyInjection/net5.0-Release/Microsoft.Extensions.DependencyInjection.dll' because it is being used by another process.  [/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.DependencyInjection/src/Microsoft.Extensions.DependencyInjection.csproj] [/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.Options/tests/TrimmingTests/Microsoft.Extensions.Options.TrimmingTests.proj]
/Users/runner/work/1/s/.dotnet/sdk/5.0.100-preview.8.20362.3/Microsoft.Common.CurrentVersion.targets(4187,5): error MSB3026: Could not copy "/Users/runner/work/1/s/artifacts/obj/Microsoft.Extensions.DependencyInjection/net5.0-Release/Microsoft.Extensions.DependencyInjection.dll" to "/Users/runner/work/1/s/artifacts/bin/Microsoft.Extensions.DependencyInjection/net5.0-Release/Microsoft.Extensions.DependencyInjection.dll". Beginning retry 1 in 1000ms. The process cannot access the file '/Users/runner/work/1/s/artifacts/obj/Microsoft.Extensions.DependencyInjection/net5.0-Release/Microsoft.Extensions.DependencyInjection.dll' because it is being used by another process.  [/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.DependencyInjection/src/Microsoft.Extensions.DependencyInjection.csproj] [/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.Options/tests/TrimmingTests/Microsoft.Extensions.Options.TrimmingTests.proj]
/Users/runner/work/1/s/eng/testing/linker/trimmingTests.targets(102,5): error MSB3073: The command ""/Users/runner/work/1/s/.dotnet/dotnet" build /t:PublishTrimmed /nr:false /warnaserror -p:configuration=Release " exited with code -1. [/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.Options/tests/TrimmingTests/Microsoft.Extensions.Options.TrimmingTests.proj]

It seems like 2 tests have a P2P to Microsoft.Extensions.DependencyInjection and for some reason it is trying to copy it from obj to bin at the same time.

Here is the binlog of the failure:
https://dev.azure.com/dnceng/_apis/resources/Containers/4760213/Logs_Build_OSX__x64_release_Runtime_Release?itemPath=Logs_Build_OSX__x64_release_Runtime_Release%2FRelease%2FTrimmingTests.binlog

Failed here: https://dev.azure.com/dnceng/public/_build/results?buildId=758123&view=logs&j=94422687-3fa3-516b-cf3b-23eb810befab&t=0015de85-1e1c-5c80-f8e6-95054477de01

cc: @eerhardt @joperezr

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

[eerhardt]

@safern and I looked a little bit at this issue last week. The issue appears that the Microsoft.Extensions.* assemblies are being referenced by the trimming tests through a ProjectReference. These projects reference other System.* assemblies (for example System.Diagnostics.DiagnosticTrace). When multiple trimming tests are being published at the same time, we are getting a race condition in building these common projects at the same time. Errors include "copying assembly from obj to bin fails because the file is in use" and sometimes MSBuild fails because of an unexpected end of stream in a .cache file.

Part of the issue is that we are shelling out to a new dotnet.exe process to publish the trimming tests. One idea to fix this would be to use the current MSBuild process instead, and just call the /t:PublishTrimmed directly through MSBuild. Another thought is that incrementalism should be kicking in here, since we've already built the product. A problem we saw is that the first build of the product is just building for net5.0, but when building the trimming tests, which require Microsoft.Extensions.* is that most of those projects don't use net5.0, and instead are netstandard projects. So the System.* references are being restored and built again for netstandard.

I've checked in a fix to these tests to disable parallelism:

https://github.com/dotnet/runtime/pull/40431/files#diff-41f0c39b7d66df4ce8506f11bbb3ee08R42

So removing blocking-clean-ci since CI should no longer fail for this issue, and moving to 6.0.

Leaving this issue open to track the underlying problem and re-enable parallelism in these tests.

cc @joperezr