Memory blowing up during dotnet/dotnet repo build

[janvorli]

Re-Bootstrap .NET to release/11-preview2 has revealed an issue in the dotnet/dotnet repo build caused by extreme memory usage leading to the build failing on Linux and for some folks even crashing WSL2 instance.
It was working fine with release/11-preview1 SDK used for build.
I was able to identify the problematic change thanks to the fact that runtime files in the 11-preview2 SDK can be patched with files from the 11-preview1 runtime build. So I've ran git bisect and found the problem was introduced in #124132.

Here are the repro steps:

• Check out https://github.com/dotnet/dotnet repo, branch release-pr-11.0.100-preview.2.26117.112. Check if the topmost commit is 41b700bc3a41fefac457cfbe316f97cf198f7c24, it is possible that more commits will get added to that branch.
• Run build.sh in the root of the repo. During the build, you'll see errors like "process killed due to signal 137" or "Child node XYZ exited prematurely"
• Please note that even in the good state, the build has two failures - one building something with fsharp and the other when building efcore, but these failures are unrelated.

To test a fix:

• Check out https://github.com/dotnet/runtime repo, branch `release/11.0-preview2.
• Make your changes there
• Build the runtime (e.g. ./build.sh clr+libs -c Release -rc Release -keepNativeSymbols true)
• Copy artifacts/bin/coreclr/linux-x64.Release/*.so and artifacts/bin/coreclr/linux-x64.Release/System.Private.CoreLib.dll to the dotnet/dotnet repo, folder .dotnet/shared/Microsoft.NETCore.App/11.0.0-preview.2.26117.112
• Switch to the dotnet/dotnet repo directory
• Run rm -r artifacts there
• Run ./build.sh

You should only see the two errors I've mentioned if your change fixes the problem.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[AndyAyersMS]

I was able to repro this but haven't yet drilled in. The failing process is running msbuild:

03:06:36     1000     52835  15055.00      0.00 148475548 6859436  10.42  /home/andy/repos/dotnet/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/andy/repos/dotnet/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj /bl:"/home/andy/repos/dotnet/src/razor/artifacts/log/Release/Build.binlog" /p:Configuration=Release /p:RepoRoot=/home/andy/repos/dotnet/src/razor/ /p:Restore=true /p:Build=true /p:DotNetBuild=true /p:DotNetBuildSourceOnly=false /p:DotNetBuildFromVMR=true /p:Rebuild=false /p:Test=false /p:Pack=true /p:IntegrationTest=false /p:PerformanceTest=false /p:Sign=false /p:Publish=true /p:RestoreStaticGraphEnableBinaryLogger=true /p:TargetRid=linux-x64 /p:DotNetPackageVersionPropsPath=/home/andy/repos/dotnet/artifacts/obj/PackageVersions/PackageVersions.razor.props /p:SourceBuiltAssetsDir=/home/andy/repos/dotnet/artifacts/assets/Release/ /p:SourceBuiltAssetManifestsDir=/home/andy/repos/dotnet/artifacts/obj/manifests/Release/razor/ /p:RepositoryName=razor /p:RepositoryUrl=https://github.com/dotnet/dotnet /p:TargetArchitecture=x64 /p:TargetOS=linux /p:SourceBuiltShippingPackagesDir=/home/andy/repos/dotnet/artifacts/packages/Release/Shipping/razor/ /p:SourceBuiltNonShippingPackagesDir=/home/andy/repos/dotnet/artifacts/packages/Release/NonShipping/razor/ /p:SourceBuiltPdbArtifactsDir=/home/andy/repos/dotnet/artifacts/SymStore/Release/razor/ /p:SourceBuiltArtifactsBasePath=/home/andy/repos/dotnet/ /p:UsingToolMicrosoftNetCompilers=false /p:PortableBuild=true /p:PortableTargetRid=linux-x64 /p:RepoOriginalSourceRevisionId=b7860660581174f32d28e3ce127f4818055dd297

[akoeplinger]

linking the revert PR: #124928

[janvorli]

I wanted to mention that the memory growth might also be caused by the native heap fragmentation. We've seen similar things happen due to OpenSSL allocation patterns where the native heap was several gigabytes large, kept growing, but the actual heap statistics were showing that 99% of that space was "free". So maybe the allocation the reverted change is doing somehow causes such fragmentation. Unfortunately, the windbg plugin to dump Linux native heap isn't working with the dump I've taken, so I cannot confirm it.

[AndyAyersMS]

I did an SPMI capture of the build, and modified the jit to print if it allocated more than 10MB..

#012844 allocated     11272192 bytes
#006540 allocated     11403264 bytes
#007496 allocated     11141120 bytes
#007347 allocated     11730944 bytes
#008093 allocated     11665408 bytes
#010022 allocated     11075584 bytes
#009852 allocated     12648448 bytes
#012096 allocated     11075584 bytes
#006468 allocated     11272192 bytes
#006451 allocated     11141120 bytes
#005787 allocated     11141120 bytes
#007187 allocated     11665408 bytes
#014328 allocated     11206656 bytes
#016570 allocated     10747904 bytes
#015612 allocated     11010048 bytes
#014081 allocated     15400960 bytes
#010990 allocated     11141120 bytes
#011731 allocated     11010048 bytes
#007584 allocated     11730944 bytes
#006604 allocated     11862016 bytes
#011844 allocated     11010048 bytes
#012649 allocated     15400960 bytes
#015779 allocated     12058624 bytes
#013672 allocated     10551296 bytes
#013196 allocated     11534336 bytes
#007565 allocated     11272192 bytes
#012238 allocated     15400960 bytes
#011824 allocated     11075584 bytes
#006544 allocated     11534336 bytes
#011638 allocated   6471811072 bytes
#011867 allocated     11010048 bytes
#005627 allocated     11993088 bytes
#011581 allocated     11796480 bytes
#006820 allocated     11534336 bytes
#006053 allocated     11534336 bytes

Looks like we've found an example where things go haywire...

With a checked jit:

#11638 (a84b8580) allocated 6473252864 bytes: NuGet.Versioning.NuGetVersion:TryGetNormalizedVersion(System.String,byref):bool