Test failure readytorun/tests/mainv1/mainv1, mainv2, BasicTestWithMcj.sh

[VincentBu]

Run: runtime-coreclr gcstress-extra 20210613.1

Failed tests:

CoreCLR Linux arm Checked gcstress0xc_jitstress1 @ (Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440

- readytorun/tests/mainv1/mainv1.sh

Assert failure(PID 5856 [0x000016e0], Thread: 5879 [0x16f7]): bIsRealStartFrameUnchanged
File: /__w/1/s/src/coreclr/vm/stackwalk.cpp Line: 1524
Image: /root/helix/work/correlation/corerun

/root/helix/work/workitem/readytorun/tests/mainv1/mainv1.sh: line 174:  5856 Aborted                 (core dumped) $CORE_ROOT/corerun $CORE_ROOT/crossgen2/crossgen2.dll -r:$CORE_ROOT/*.dll -r:pwd -o:mainv1.ni.dll mainv1.dll

Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/Reports/readytorun.tests/mainv1/mainv1.output.txt
Raw output:
BEGIN EXECUTION
Warning: No files matching /root/helix/work/workitem/readytorun/tests/mainv1
Emitting R2R PE file: test.ni.dll
Warning: No files matching /root/helix/work/workitem/readytorun/tests/mainv1
Emitting R2R PE file: fieldgetter.ni.dll
Warning: No files matching /root/helix/work/workitem/readytorun/tests/mainv1
Gathering state for process 5856 corerun
Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.5856.dmp
Written 127619072 bytes (31157 pages) to core file
Dump successfully written
Crossgen failed with exitcode: 134
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=/root/helix/work/correlation
/root/helix/work/workitem/readytorun/tests/mainv1/mainv1.sh
Expected: True
Actual:   False


Stack trace
   at readytorun_tests._mainv1_mainv1_._mainv1_mainv1_sh() in readytorun.tests.XUnitWrapper.dll:token 0x600000c+0x295

CoreCLR Linux arm Checked gcstress0xc_tailcallstress @ (Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440

- readytorun/tests/mainv1/mainv1.sh

Assert failure(PID 5803 [0x000016ab], Thread: 5819 [0x16bb]): (GetComponentSize() <= 2) || IsArray()
File: /__w/1/s/src/coreclr/vm/methodtable.cpp Line: 7519
Image: /root/helix/work/correlation/corerun

/root/helix/work/workitem/readytorun/tests/mainv1/mainv1.sh: line 160:  5803 Aborted                 (core dumped) $CORE_ROOT/corerun $CORE_ROOT/crossgen2/crossgen2.dll -r:$CORE_ROOT/*.dll -r:pwd -o:fieldgetter.ni.dll fieldgetter.dll

Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/Reports/readytorun.tests/mainv1/mainv1.output.txt
Raw output:
BEGIN EXECUTION
Warning: No files matching /root/helix/work/workitem/readytorun/tests/mainv1
Emitting R2R PE file: test.ni.dll
Warning: No files matching /root/helix/work/workitem/readytorun/tests/mainv1
Gathering state for process 5803 corerun
Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.5803.dmp
Written 114253824 bytes (27894 pages) to core file
Dump successfully written
Crossgen failed with exitcode: 134
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=/root/helix/work/correlation
/root/helix/work/workitem/readytorun/tests/mainv1/mainv1.sh
Expected: True
Actual:   False


Stack trace
   at readytorun_tests._mainv1_mainv1_._mainv1_mainv1_sh() in readytorun.tests.XUnitWrapper.dll:token 0x600000c+0x295

CoreCLR Linux arm64 Checked gcstress0xc_jitstress1 @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652

- readytorun/tests/mainv1/mainv1.sh
- readytorun/tests/mainv2/mainv2.sh
- baseservices/TieredCompilation/BasicTestWithMcj/BasicTestWithMcj.sh

Unhandled exception. System.IndexOutOfRangeException: Index was outside the bounds of the array.
at System.Reflection.Metadata.Ecma335.MetadataSizes.ReferenceFits(Int32 bitCount, TableIndex[] tables) in //src/libraries/System.Reflection.Metadata/src/System/Reflection/Metadata/Ecma335/MetadataSizes.cs:line 472
at System.Reflection.Metadata.Ecma335.MetadataSizes..ctor(ImmutableArray1 rowCounts, ImmutableArray1 externalRowCounts, ImmutableArray1 heapSizes, Int32 metadataVersionByteCount, Boolean isStandaloneDebugMetadata) in /_/src/libraries/System.Reflection.Metadata/src/System/Reflection/Metadata/Ecma335/MetadataSizes.cs:line 104 at System.Reflection.Metadata.Ecma335.MetadataBuilder.GetSerializedMetadata(ImmutableArray1 externalRowCounts, Int32 metadataVersionByteCount, Boolean isStandaloneDebugMetadata) in //src/libraries/System.Reflection.Metadata/src/System/Reflection/Metadata/Ecma335/MetadataBuilder.cs:line 22
at System.Reflection.Metadata.Ecma335.MetadataRootBuilder..ctor(MetadataBuilder tablesAndHeaps, String metadataVersion, Boolean suppressValidation) in /_/src/libraries/System.Reflection.Metadata/src/System/Reflection/Metadata/Ecma335/MetadataRootBuilder.cs:line 71
at ILCompiler.DependencyAnalysis.ReadyToRun.ManifestMetadataTableNode.GetData(NodeFactory factory, Boolean relocsOnly) in ILCompiler.ReadyToRun.dll:token 0x6000562+0x251
at ILCompiler.DependencyAnalysis.ReadyToRunObjectWriter.EmitPortableExecutable() in ILCompiler.ReadyToRun.dll:token 0x6000324+0x174
at ILCompiler.ReadyToRunCodegenCompilation.Compile(String outputFile) in ILCompiler.ReadyToRun.dll:token 0x6000130+0x35
at ILCompiler.Program.RunSingleCompilation(Dictionary2 inFilePaths, InstructionSetSupport instructionSetSupport, String compositeRootPath, Dictionary2 unrootedInputFilePaths, HashSet1 versionBubbleModulesHash, CompilerTypeSystemContext typeSystemContext) in crossgen2.dll:token 0x600015a+0x5f6 at ILCompiler.Program.Run(String[] args) in crossgen2.dll:token 0x6000159+0x52e at ILCompiler.Program.Main(String[] args) in crossgen2.dll:token 0x6000161+0x5 apply_reg_state: ip and cfa unchanged; stopping here (ip=0x7fa2adad0c) /root/helix/work/workitem/readytorun/tests/mainv1/mainv1.sh: line 146: 3493 Aborted (core dumped) $CORE_ROOT/corerun $CORE_ROOT/crossgen2/crossgen2.dll -r:$CORE_ROOT/*.dll -r:pwd` -o:test.ni.dll test.dll

Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/Reports/readytorun.tests/mainv1/mainv1.output.txt
Raw output:
BEGIN EXECUTION
Warning: No files matching /root/helix/work/workitem/readytorun/tests/mainv1
Emitting R2R PE file: test.ni.dll
Gathering state for process 3493 corerun
Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.3493.dmp
Written 137220096 bytes (33501 pages) to core file
Dump successfully written
Crossgen failed with exitcode: 134
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=/root/helix/work/correlation
/root/helix/work/workitem/readytorun/tests/mainv1/mainv1.sh
Expected: True
Actual:   False


Stack trace
   at readytorun_tests._mainv1_mainv1_._mainv1_mainv1_sh() in readytorun.tests.XUnitWrapper.dll:token 0x600000c+0x295

[VincentBu]

Historical failures of this test:

Queued	OS	Arch	Pipeline
2021-06-14T04:35:20.295Z	ubuntu.1804.armarch.open	arm	runtime-coreclr r2r-extra Checked-gcstress0xf
2021-06-14T04:17:37.939Z	ubuntu.1804.amd64.open.rt	x64	runtime-coreclr r2r-extra Checked-gcstress0xf
2021-06-14T00:36:47.473Z	ubuntu.1804.armarch.open	arm	runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-06-13T23:32:05.639Z	ubuntu.1804.armarch.open	arm64	runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-06-13T23:13:12.59Z	ubuntu.1804.armarch.open	arm	runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-06-13T22:37:43.533Z	osx.1100.arm64.open	arm64	runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-06-13T22:31:51.626Z	ubuntu.1804.armarch.open	arm64	runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-06-13T14:20:22.387Z	ubuntu.1804.armarch.open	arm64	runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-06-13T14:19:50.897Z	ubuntu.1804.armarch.open	arm	runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3

More failures

Queued	OS	Arch	Pipeline
2021-06-13T14:19:26.263Z	ubuntu.1804.amd64.open.rt	x64	runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-06-13T07:11:53.836Z	ubuntu.1804.armarch.open	arm64	runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_jitstress2
2021-06-13T05:05:24.995Z	ubuntu.1804.armarch.open	arm	runtime-coreclr r2r-extra Checked-gcstress0xf
2021-06-13T04:38:30.643Z	ubuntu.1804.amd64.open.rt	x64	runtime-coreclr r2r-extra Checked-gcstress0xf
2021-06-12T23:23:37.045Z	ubuntu.1804.armarch.open	arm64	runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-06-12T22:21:09.913Z	ubuntu.1804.armarch.open	arm	runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-06-12T14:18:18.372Z	osx.1100.arm64.open	arm64	runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-06-12T14:17:55.352Z	ubuntu.1804.armarch.open	arm64	runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-06-12T14:17:02.579Z	ubuntu.1804.amd64.open.rt	x64	runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-06-12T14:16:32.329Z	ubuntu.1804.armarch.open	arm	runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-06-12T14:16:30.674Z	ubuntu.1804.armarch.open	arm	runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-06-07T09:03:35.729Z	osx.1100.arm64.open	arm64	runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress2
2021-05-31T08:55:01.733Z	osx.1100.arm64.open	arm64	runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1
2021-05-30T08:49:30.455Z	osx.1100.arm64.open	arm64	runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1
2021-05-24T08:38:49.433Z	osx.1100.arm64.open	arm64	runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1
2021-05-16T07:50:06.006Z	osx.1100.arm64.open	arm64	runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_jitstress2
2021-05-16T07:41:51.341Z	ubuntu.1804.armarch.open	arm64	runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_jitstress2
2021-05-10T07:53:46.602Z	ubuntu.1804.armarch.open	arm64	runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_jitstress2
2021-05-09T09:26:35.32Z	osx.1100.arm64.open	arm64	runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1

[BruceForstall]

I see mainv1 and mainv2 continue to fail in GCStress, e.g. GCStress=3 for Linux/x64 and Linux/arm:

https://dev.azure.com/dnceng/public/_build/results?buildId=1196527&view=ms.vss-test-web.build-test-results-tab&runId=35881476&resultId=110466&paneView=debug

In that case, it looks like a timeout, but I wonder if that's inaccurate.

Lots of additional GCStress failures here:

https://dev.azure.com/dnceng/public/_build/results?buildId=1196682&view=ms.vss-test-web.build-test-results-tab&runId=35883968&resultId=111047&paneView=debug

[BruceForstall]

@dotnet/crossgen-contrib